Skip to content

dataflow.v1beta1.job

"Job is the Schema for the Jobs API. Creates a job in Dataflow according to a provided config file."

Index

Fields

fn new

new(name)

new returns an instance of Job

obj metadata

"ObjectMeta is metadata that all persisted resources must have, which includes all objects users must create."

fn metadata.withAnnotations

withAnnotations(annotations)

"Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: http://kubernetes.io/docs/user-guide/annotations"

fn metadata.withAnnotationsMixin

withAnnotationsMixin(annotations)

"Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: http://kubernetes.io/docs/user-guide/annotations"

Note: This function appends passed data to existing values

fn metadata.withClusterName

withClusterName(clusterName)

"The name of the cluster which the object belongs to. This is used to distinguish resources with same name and namespace in different clusters. This field is not set anywhere right now and apiserver is going to ignore it if set in create or update request."

fn metadata.withCreationTimestamp

withCreationTimestamp(creationTimestamp)

"Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers."

fn metadata.withDeletionGracePeriodSeconds

withDeletionGracePeriodSeconds(deletionGracePeriodSeconds)

"Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only."

fn metadata.withDeletionTimestamp

withDeletionTimestamp(deletionTimestamp)

"Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers."

fn metadata.withFinalizers

withFinalizers(finalizers)

"Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list."

fn metadata.withFinalizersMixin

withFinalizersMixin(finalizers)

"Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list."

Note: This function appends passed data to existing values

fn metadata.withGenerateName

withGenerateName(generateName)

"GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.\n\nIf this field is specified and the generated name exists, the server will NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header).\n\nApplied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency"

fn metadata.withGeneration

withGeneration(generation)

"A sequence number representing a specific generation of the desired state. Populated by the system. Read-only."

fn metadata.withLabels

withLabels(labels)

"Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: http://kubernetes.io/docs/user-guide/labels"

fn metadata.withLabelsMixin

withLabelsMixin(labels)

"Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: http://kubernetes.io/docs/user-guide/labels"

Note: This function appends passed data to existing values

fn metadata.withName

withName(name)

"Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names"

fn metadata.withNamespace

withNamespace(namespace)

"Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the \"default\" namespace, but \"default\" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.\n\nMust be a DNS_LABEL. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/namespaces"

fn metadata.withOwnerReferences

withOwnerReferences(ownerReferences)

"List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller."

fn metadata.withOwnerReferencesMixin

withOwnerReferencesMixin(ownerReferences)

"List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller."

Note: This function appends passed data to existing values

fn metadata.withResourceVersion

withResourceVersion(resourceVersion)

"An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.\n\nPopulated by the system. Read-only. Value must be treated as opaque by clients and . More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency"

withSelfLink(selfLink)

"SelfLink is a URL representing this object. Populated by the system. Read-only.\n\nDEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release."

fn metadata.withUid

withUid(uid)

"UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.\n\nPopulated by the system. Read-only. More info: http://kubernetes.io/docs/user-guide/identifiers#uids"

obj spec

"JobSpec defines the desired state of Job"

fn spec.withDeletionPolicy

withDeletionPolicy(deletionPolicy)

"DeletionPolicy specifies what will happen to the underlying external when this managed resource is deleted - either \"Delete\" or \"Orphan\" the external resource. This field is planned to be deprecated in favor of the ManagementPolicies field in a future release. Currently, both could be set independently and non-default values would be honored if the feature flag is enabled. See the design doc for more information: https://github.com/crossplane/crossplane/blob/499895a25d1a1a0ba1604944ef98ac7a1a71f197/design/design-doc-observe-only-resources.md?plain=1#L223"

fn spec.withManagementPolicies

withManagementPolicies(managementPolicies)

"THIS IS AN ALPHA FIELD. Do not use it in production. It is not honored unless the relevant Crossplane feature flag is enabled, and may be changed or removed without notice. ManagementPolicies specify the array of actions Crossplane is allowed to take on the managed and external resources. This field is planned to replace the DeletionPolicy field in a future release. Currently, both could be set independently and non-default values would be honored if the feature flag is enabled. If both are custom, the DeletionPolicy field will be ignored. See the design doc for more information: https://github.com/crossplane/crossplane/blob/499895a25d1a1a0ba1604944ef98ac7a1a71f197/design/design-doc-observe-only-resources.md?plain=1#L223 and this one: https://github.com/crossplane/crossplane/blob/444267e84783136daa93568b364a5f01228cacbe/design/one-pager-ignore-changes.md"

fn spec.withManagementPoliciesMixin

withManagementPoliciesMixin(managementPolicies)

"THIS IS AN ALPHA FIELD. Do not use it in production. It is not honored unless the relevant Crossplane feature flag is enabled, and may be changed or removed without notice. ManagementPolicies specify the array of actions Crossplane is allowed to take on the managed and external resources. This field is planned to replace the DeletionPolicy field in a future release. Currently, both could be set independently and non-default values would be honored if the feature flag is enabled. If both are custom, the DeletionPolicy field will be ignored. See the design doc for more information: https://github.com/crossplane/crossplane/blob/499895a25d1a1a0ba1604944ef98ac7a1a71f197/design/design-doc-observe-only-resources.md?plain=1#L223 and this one: https://github.com/crossplane/crossplane/blob/444267e84783136daa93568b364a5f01228cacbe/design/one-pager-ignore-changes.md"

Note: This function appends passed data to existing values

obj spec.forProvider

fn spec.forProvider.withAdditionalExperiments

withAdditionalExperiments(additionalExperiments)

"List of experiments that should be used by the job. An example value is [\"enable_stackdriver_agent_metrics\"]."

fn spec.forProvider.withAdditionalExperimentsMixin

withAdditionalExperimentsMixin(additionalExperiments)

"List of experiments that should be used by the job. An example value is [\"enable_stackdriver_agent_metrics\"]."

Note: This function appends passed data to existing values

fn spec.forProvider.withEnableStreamingEngine

withEnableStreamingEngine(enableStreamingEngine)

"Enable/disable the use of Streaming Engine for the job. Note that Streaming Engine is enabled by default for pipelines developed against the Beam SDK for Python v2.21.0 or later when using Python 3."

fn spec.forProvider.withIpConfiguration

withIpConfiguration(ipConfiguration)

"The configuration for VM IPs. Options are \"WORKER_IP_PUBLIC\" or \"WORKER_IP_PRIVATE\"."

fn spec.forProvider.withKmsKeyName

withKmsKeyName(kmsKeyName)

"The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY"

fn spec.forProvider.withLabels

withLabels(labels)

"User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply."

fn spec.forProvider.withLabelsMixin

withLabelsMixin(labels)

"User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply."

Note: This function appends passed data to existing values

fn spec.forProvider.withMachineType

withMachineType(machineType)

"The machine type to use for the job."

fn spec.forProvider.withMaxWorkers

withMaxWorkers(maxWorkers)

"The number of workers permitted to work on the job. More workers may improve processing speed at additional cost."

fn spec.forProvider.withName

withName(name)

"A unique name for the resource, required by Dataflow."

fn spec.forProvider.withNetwork

withNetwork(network)

"The network to which VMs will be assigned. If it is not provided, \"default\" will be used."

fn spec.forProvider.withOnDelete

withOnDelete(onDelete)

"One of \"drain\" or \"cancel\". See above note."

fn spec.forProvider.withParameters

withParameters(parameters)

"Key/Value pairs to be passed to the Dataflow job (as used in the template)."

fn spec.forProvider.withParametersMixin

withParametersMixin(parameters)

"Key/Value pairs to be passed to the Dataflow job (as used in the template)."

Note: This function appends passed data to existing values

fn spec.forProvider.withProject

withProject(project)

"The project in which the resource belongs. If it is not provided, the provider project is used."

fn spec.forProvider.withRegion

withRegion(region)

"The region in which the created job should run."

fn spec.forProvider.withServiceAccountEmail

withServiceAccountEmail(serviceAccountEmail)

"The Service Account email used to create the job."

fn spec.forProvider.withSkipWaitOnJobTermination

withSkipWaitOnJobTermination(skipWaitOnJobTermination)

"See above note."

fn spec.forProvider.withSubnetwork

withSubnetwork(subnetwork)

"The subnetwork to which VMs will be assigned. Should be of the form \"regions/REGION/subnetworks/SUBNETWORK\". If the subnetwork is located in a Shared VPC network, you must use the complete URL. For example \"googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/subnetworks/SUBNET_NAME\

fn spec.forProvider.withTempGcsLocation

withTempGcsLocation(tempGcsLocation)

"A writeable location on GCS for the Dataflow job to dump its temporary data."

fn spec.forProvider.withTemplateGcsPath

withTemplateGcsPath(templateGcsPath)

"The GCS path to the Dataflow job template."

fn spec.forProvider.withTransformNameMapping

withTransformNameMapping(transformNameMapping)

"Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job. This field is not used outside of update."

fn spec.forProvider.withTransformNameMappingMixin

withTransformNameMappingMixin(transformNameMapping)

"Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job. This field is not used outside of update."

Note: This function appends passed data to existing values

fn spec.forProvider.withZone

withZone(zone)

"The zone in which the created job should run. If it is not provided, the provider zone is used."

obj spec.initProvider

"THIS IS AN ALPHA FIELD. Do not use it in production. It is not honored unless the relevant Crossplane feature flag is enabled, and may be changed or removed without notice. InitProvider holds the same fields as ForProvider, with the exception of Identifier and other resource reference fields. The fields that are in InitProvider are merged into ForProvider when the resource is created. The same fields are also added to the terraform ignore_changes hook, to avoid updating them after creation. This is useful for fields that are required on creation, but we do not desire to update them after creation, for example because of an external controller is managing them, like an autoscaler."

fn spec.initProvider.withAdditionalExperiments

withAdditionalExperiments(additionalExperiments)

"List of experiments that should be used by the job. An example value is [\"enable_stackdriver_agent_metrics\"]."

fn spec.initProvider.withAdditionalExperimentsMixin

withAdditionalExperimentsMixin(additionalExperiments)

"List of experiments that should be used by the job. An example value is [\"enable_stackdriver_agent_metrics\"]."

Note: This function appends passed data to existing values

fn spec.initProvider.withEnableStreamingEngine

withEnableStreamingEngine(enableStreamingEngine)

"Enable/disable the use of Streaming Engine for the job. Note that Streaming Engine is enabled by default for pipelines developed against the Beam SDK for Python v2.21.0 or later when using Python 3."

fn spec.initProvider.withIpConfiguration

withIpConfiguration(ipConfiguration)

"The configuration for VM IPs. Options are \"WORKER_IP_PUBLIC\" or \"WORKER_IP_PRIVATE\"."

fn spec.initProvider.withKmsKeyName

withKmsKeyName(kmsKeyName)

"The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY"

fn spec.initProvider.withLabels

withLabels(labels)

"User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply."

fn spec.initProvider.withLabelsMixin

withLabelsMixin(labels)

"User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply."

Note: This function appends passed data to existing values

fn spec.initProvider.withMachineType

withMachineType(machineType)

"The machine type to use for the job."

fn spec.initProvider.withMaxWorkers

withMaxWorkers(maxWorkers)

"The number of workers permitted to work on the job. More workers may improve processing speed at additional cost."

fn spec.initProvider.withName

withName(name)

"A unique name for the resource, required by Dataflow."

fn spec.initProvider.withNetwork

withNetwork(network)

"The network to which VMs will be assigned. If it is not provided, \"default\" will be used."

fn spec.initProvider.withOnDelete

withOnDelete(onDelete)

"One of \"drain\" or \"cancel\". See above note."

fn spec.initProvider.withParameters

withParameters(parameters)

"Key/Value pairs to be passed to the Dataflow job (as used in the template)."

fn spec.initProvider.withParametersMixin

withParametersMixin(parameters)

"Key/Value pairs to be passed to the Dataflow job (as used in the template)."

Note: This function appends passed data to existing values

fn spec.initProvider.withProject

withProject(project)

"The project in which the resource belongs. If it is not provided, the provider project is used."

fn spec.initProvider.withRegion

withRegion(region)

"The region in which the created job should run."

fn spec.initProvider.withServiceAccountEmail

withServiceAccountEmail(serviceAccountEmail)

"The Service Account email used to create the job."

fn spec.initProvider.withSkipWaitOnJobTermination

withSkipWaitOnJobTermination(skipWaitOnJobTermination)

"See above note."

fn spec.initProvider.withSubnetwork

withSubnetwork(subnetwork)

"The subnetwork to which VMs will be assigned. Should be of the form \"regions/REGION/subnetworks/SUBNETWORK\". If the subnetwork is located in a Shared VPC network, you must use the complete URL. For example \"googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/subnetworks/SUBNET_NAME\

fn spec.initProvider.withTempGcsLocation

withTempGcsLocation(tempGcsLocation)

"A writeable location on GCS for the Dataflow job to dump its temporary data."

fn spec.initProvider.withTemplateGcsPath

withTemplateGcsPath(templateGcsPath)

"The GCS path to the Dataflow job template."

fn spec.initProvider.withTransformNameMapping

withTransformNameMapping(transformNameMapping)

"Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job. This field is not used outside of update."

fn spec.initProvider.withTransformNameMappingMixin

withTransformNameMappingMixin(transformNameMapping)

"Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job. This field is not used outside of update."

Note: This function appends passed data to existing values

fn spec.initProvider.withZone

withZone(zone)

"The zone in which the created job should run. If it is not provided, the provider zone is used."

obj spec.providerConfigRef

"ProviderConfigReference specifies how the provider that will be used to create, observe, update, and delete this managed resource should be configured."

fn spec.providerConfigRef.withName

withName(name)

"Name of the referenced object."

obj spec.providerConfigRef.policy

"Policies for referencing."

fn spec.providerConfigRef.policy.withResolution

withResolution(resolution)

"Resolution specifies whether resolution of this reference is required. The default is 'Required', which means the reconcile will fail if the reference cannot be resolved. 'Optional' means this reference will be a no-op if it cannot be resolved."

fn spec.providerConfigRef.policy.withResolve

withResolve(resolve)

"Resolve specifies when this reference should be resolved. The default is 'IfNotPresent', which will attempt to resolve the reference only when the corresponding field is not present. Use 'Always' to resolve the reference on every reconcile."

obj spec.providerRef

"ProviderReference specifies the provider that will be used to create, observe, update, and delete this managed resource. Deprecated: Please use ProviderConfigReference, i.e. providerConfigRef"

fn spec.providerRef.withName

withName(name)

"Name of the referenced object."

obj spec.providerRef.policy

"Policies for referencing."

fn spec.providerRef.policy.withResolution

withResolution(resolution)

"Resolution specifies whether resolution of this reference is required. The default is 'Required', which means the reconcile will fail if the reference cannot be resolved. 'Optional' means this reference will be a no-op if it cannot be resolved."

fn spec.providerRef.policy.withResolve

withResolve(resolve)

"Resolve specifies when this reference should be resolved. The default is 'IfNotPresent', which will attempt to resolve the reference only when the corresponding field is not present. Use 'Always' to resolve the reference on every reconcile."

obj spec.publishConnectionDetailsTo

"PublishConnectionDetailsTo specifies the connection secret config which contains a name, metadata and a reference to secret store config to which any connection details for this managed resource should be written. Connection details frequently include the endpoint, username, and password required to connect to the managed resource."

fn spec.publishConnectionDetailsTo.withName

withName(name)

"Name is the name of the connection secret."

obj spec.publishConnectionDetailsTo.configRef

"SecretStoreConfigRef specifies which secret store config should be used for this ConnectionSecret."

fn spec.publishConnectionDetailsTo.configRef.withName

withName(name)

"Name of the referenced object."

obj spec.publishConnectionDetailsTo.configRef.policy

"Policies for referencing."

fn spec.publishConnectionDetailsTo.configRef.policy.withResolution

withResolution(resolution)

"Resolution specifies whether resolution of this reference is required. The default is 'Required', which means the reconcile will fail if the reference cannot be resolved. 'Optional' means this reference will be a no-op if it cannot be resolved."

fn spec.publishConnectionDetailsTo.configRef.policy.withResolve

withResolve(resolve)

"Resolve specifies when this reference should be resolved. The default is 'IfNotPresent', which will attempt to resolve the reference only when the corresponding field is not present. Use 'Always' to resolve the reference on every reconcile."

obj spec.publishConnectionDetailsTo.metadata

"Metadata is the metadata for connection secret."

fn spec.publishConnectionDetailsTo.metadata.withAnnotations

withAnnotations(annotations)

"Annotations are the annotations to be added to connection secret. - For Kubernetes secrets, this will be used as \"metadata.annotations\". - It is up to Secret Store implementation for others store types."

fn spec.publishConnectionDetailsTo.metadata.withAnnotationsMixin

withAnnotationsMixin(annotations)

"Annotations are the annotations to be added to connection secret. - For Kubernetes secrets, this will be used as \"metadata.annotations\". - It is up to Secret Store implementation for others store types."

Note: This function appends passed data to existing values

fn spec.publishConnectionDetailsTo.metadata.withLabels

withLabels(labels)

"Labels are the labels/tags to be added to connection secret. - For Kubernetes secrets, this will be used as \"metadata.labels\". - It is up to Secret Store implementation for others store types."

fn spec.publishConnectionDetailsTo.metadata.withLabelsMixin

withLabelsMixin(labels)

"Labels are the labels/tags to be added to connection secret. - For Kubernetes secrets, this will be used as \"metadata.labels\". - It is up to Secret Store implementation for others store types."

Note: This function appends passed data to existing values

fn spec.publishConnectionDetailsTo.metadata.withType

withType(type)

"Type is the SecretType for the connection secret. - Only valid for Kubernetes Secret Stores."

obj spec.writeConnectionSecretToRef

"WriteConnectionSecretToReference specifies the namespace and name of a Secret to which any connection details for this managed resource should be written. Connection details frequently include the endpoint, username, and password required to connect to the managed resource. This field is planned to be replaced in a future release in favor of PublishConnectionDetailsTo. Currently, both could be set independently and connection details would be published to both without affecting each other."

fn spec.writeConnectionSecretToRef.withName

withName(name)

"Name of the secret."

fn spec.writeConnectionSecretToRef.withNamespace

withNamespace(namespace)

"Namespace of the secret."