AWS Glue using job bookmark fails with "Datasource does not support writing empty or nested empty schemas"

Question

I am working in a Glue (version 2.0) job using the bookmark feature, in the second time that I run the job without any changes on the file, I get the following error message:

AnalysisException: '\nDatasource does not support writing empty or nested empty schemas.\nPlease make sure the data schema has at least one or more column(s).\n ;'

It is a generated script by AWS console without any modifications, the source is S3 files using data catalog and the output is another bucket.

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
## @type: DataSource
## @args: [database = "raw_dev_edocs", table_name = "esocial_s2200", transformation_ctx = "datasource0"]
## @return: datasource0
## @inputs: []
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "raw_dev_edocs", table_name = "esocial_s2200", transformation_ctx = "datasource0")
## @type: ApplyMapping
## @args: [mapping = [("esocial", "struct", "esocial", "struct"), ("tenant", "string", "tenant", "string"), ("year", "string", "year", "string"), ("month", "string", "month", "string"), ("day", "string", "day", "string")], transformation_ctx = "applymapping1"]
## @return: applymapping1
## @inputs: [frame = datasource0]
applymapping1 = ApplyMapping.apply(frame = datasource0, mappings = [("esocial", "struct", "esocial", "struct"), ("tenant", "string", "tenant", "string"), ("year", "string", "year", "string"), ("month", "string", "month", "string"), ("day", "string", "day", "string")], transformation_ctx = "applymapping1")
## @type: ResolveChoice
## @args: [choice = "make_struct", transformation_ctx = "resolvechoice2"]
## @return: resolvechoice2
## @inputs: [frame = applymapping1]
resolvechoice2 = ResolveChoice.apply(frame = applymapping1, choice = "make_struct", transformation_ctx = "resolvechoice2")
## @type: DropNullFields
## @args: [transformation_ctx = "dropnullfields3"]
## @return: dropnullfields3
## @inputs: [frame = resolvechoice2]
dropnullfields3 = DropNullFields.apply(frame = resolvechoice2, transformation_ctx = "dropnullfields3")
## @type: DataSink
## @args: [connection_type = "s3", connection_options = {"path": "s3://senior-datalake-stage/dev/edocs/S2200test"}, format = "parquet", transformation_ctx = "datasink4"]
## @return: datasink4
## @inputs: [frame = dropnullfields3]
datasink4 = glueContext.write_dynamic_frame.from_options(frame = dropnullfields3, connection_type = "s3", connection_options = {"path": "s3://senior-datalake-stage/dev/edocs/S2200test"}, format = "parquet", transformation_ctx = "datasink4")
job.commit()

As mentioned in this doc (https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html) the bookmark keeps track using the modification timestamp, my script is the same as the example of the documentation.

When I reset the bookmark, the job run successfully.

The table schema where Glue is reading from is:

Column name
Data type
Key
Comment
1
tenant
string
Partition (0)

2
year
string
Partition (1)

3
month
string
Partition (2)

4
day
string
Partition (3)

5
esocial.xmlns
string

6
esocial.evtadmissao.id
string

7
esocial.evtadmissao.ideevento.indretif
string

8
esocial.evtadmissao.ideevento.tpamb
string

9
esocial.evtadmissao.ideevento.procemi
string

10
esocial.evtadmissao.ideevento.verproc
string

11
esocial.evtadmissao.ideevento.nrrecibo
string

12
esocial.evtadmissao.ideempregador.tpinsc
string

13
esocial.evtadmissao.ideempregador.nrinsc
string

14
esocial.evtadmissao.trabalhador.cpftrab
string

15
esocial.evtadmissao.trabalhador.nistrab
string

16
esocial.evtadmissao.trabalhador.nmtrab
string

17
esocial.evtadmissao.trabalhador.sexo
string

18
esocial.evtadmissao.trabalhador.racacor
string

19
esocial.evtadmissao.trabalhador.estciv
string

20
esocial.evtadmissao.trabalhador.grauinstr
string

21
esocial.evtadmissao.trabalhador.nascimento.dtnascto
string

22
esocial.evtadmissao.trabalhador.nascimento.codmunic
string

23
esocial.evtadmissao.trabalhador.nascimento.uf
string

24
esocial.evtadmissao.trabalhador.nascimento.paisnascto
string

25
esocial.evtadmissao.trabalhador.nascimento.paisnac
string

26
esocial.evtadmissao.trabalhador.nascimento.nmmae
string

27
esocial.evtadmissao.trabalhador.nascimento.nmpai
string

28
esocial.evtadmissao.trabalhador.documentos.ctps.nrctps
string

29
esocial.evtadmissao.trabalhador.documentos.ctps.seriectps
string

30
esocial.evtadmissao.trabalhador.documentos.ctps.ufctps
string

31
esocial.evtadmissao.trabalhador.documentos.rg.nrrg
string

32
esocial.evtadmissao.trabalhador.documentos.rg.orgaoemissor
string

33
esocial.evtadmissao.trabalhador.documentos.rg.dtexped
string

34
esocial.evtadmissao.trabalhador.documentos.rne.nrrne
string

35
esocial.evtadmissao.trabalhador.documentos.rne.orgaoemissor
string

36
esocial.evtadmissao.trabalhador.documentos.rne.dtexped
string

37
esocial.evtadmissao.trabalhador.endereco.brasil.tplograd
string

38
esocial.evtadmissao.trabalhador.endereco.brasil.dsclograd
string

39
esocial.evtadmissao.trabalhador.endereco.brasil.nrlograd
string

40
esocial.evtadmissao.trabalhador.endereco.brasil.bairro
string

41
esocial.evtadmissao.trabalhador.endereco.brasil.cep
string

42
esocial.evtadmissao.trabalhador.endereco.brasil.codmunic
string

43
esocial.evtadmissao.trabalhador.endereco.brasil.uf
string

44
esocial.evtadmissao.trabalhador.endereco.brasil.complemento
string

45
esocial.evtadmissao.trabalhador.infodeficiencia.deffisica
string

46
esocial.evtadmissao.trabalhador.infodeficiencia.defvisual
string

47
esocial.evtadmissao.trabalhador.infodeficiencia.defauditiva
string

48
esocial.evtadmissao.trabalhador.infodeficiencia.defmental
string

49
esocial.evtadmissao.trabalhador.infodeficiencia.defintelectual
string

50
esocial.evtadmissao.trabalhador.infodeficiencia.reabreadap
string

51
esocial.evtadmissao.trabalhador.infodeficiencia.infocota
string

52
esocial.evtadmissao.trabalhador.contato.foneprinc
string

53
esocial.evtadmissao.trabalhador.contato.fonealternat
string

54
esocial.evtadmissao.trabalhador.contato.emailprinc
string

55
esocial.evtadmissao.trabalhador.contato.emailalternat
string

56
esocial.evtadmissao.trabalhador.dependente.struct.tpdep
string

57
esocial.evtadmissao.trabalhador.dependente.struct.nmdep
string

58
esocial.evtadmissao.trabalhador.dependente.struct.dtnascto
string

59
esocial.evtadmissao.trabalhador.dependente.struct.cpfdep
string

60
esocial.evtadmissao.trabalhador.dependente.struct.depirrf
string

61
esocial.evtadmissao.trabalhador.dependente.struct.depsf
string

62
esocial.evtadmissao.trabalhador.dependente.struct.inctrab
string

63
esocial.evtadmissao.trabalhador.dependente.array
bigint

64
esocial.evtadmissao.trabalhador.indpriempr
string

65
esocial.evtadmissao.trabalhador.trabestrangeiro.dtchegada
string

66
esocial.evtadmissao.trabalhador.trabestrangeiro.classtrabestrang
string

67
esocial.evtadmissao.trabalhador.trabestrangeiro.casadobr
string

68
esocial.evtadmissao.trabalhador.trabestrangeiro.filhosbr
string

69
esocial.evtadmissao.vinculo.matricula
string

70
esocial.evtadmissao.vinculo.tpregtrab
string

71
esocial.evtadmissao.vinculo.tpregprev
string

72
esocial.evtadmissao.vinculo.cadini
string

73
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.dtadm
string

74
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.tpadmissao
string

75
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.indadmissao
string

76
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.tpregjor
string

77
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.natatividade
string

78
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.dtbase
string

79
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.cnpjsindcategprof
string

80
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.fgts.opcfgts
string

81
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.fgts.dtopcfgts
string

82
esocial.evtadmissao.vinculo.infocontrato.codcargo
string

83
esocial.evtadmissao.vinculo.infocontrato.codcateg
string

84
esocial.evtadmissao.vinculo.infocontrato.remuneracao.vrsalfx
string

85
esocial.evtadmissao.vinculo.infocontrato.remuneracao.undsalfixo
string

86
esocial.evtadmissao.vinculo.infocontrato.duracao.tpcontr
string

87
esocial.evtadmissao.vinculo.infocontrato.duracao.dtterm
string

88
esocial.evtadmissao.vinculo.infocontrato.duracao.clauassec
string

89
esocial.evtadmissao.vinculo.infocontrato.localtrabalho.localtrabgeral.tpinsc
string

90
esocial.evtadmissao.vinculo.infocontrato.localtrabalho.localtrabgeral.nrinsc
string

91
esocial.evtadmissao.vinculo.infocontrato.horcontratual.qtdhrssem
string

92
esocial.evtadmissao.vinculo.infocontrato.horcontratual.tpjornada
string

93
esocial.evtadmissao.vinculo.infocontrato.horcontratual.tmpparc
string

94
esocial.evtadmissao.vinculo.infocontrato.horcontratual.horario
bigint

95
esocial.evtadmissao.vinculo.infocontrato.filiacaosindical.cnpjsindtrab
string

96
esocial.evtadmissao.vinculo.infocontrato.observacoes.observacao
string

97
esocial.evtadmissao.vinculo.afastamento.dtiniafast
string

98
esocial.evtadmissao.vinculo.afastamento.codmotafast
string

Does anyone knows how can I solve this problem?

Answer 1

I did a workaround to solve the problem.

if len(dropnullfields3.select('root').toDF().schema) > 0:
    datasink4 = glueContext.write_dynamic_frame.from_options(frame = dropnullfields3, connection_type = "s3", connection_options = {"path": "s3://senior-datalake-stage/dev/edocs/S2200test"}, format = "parquet", transformation_ctx = "datasink4")
    job.commit()