René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"docker-compose deployment of v0.6-branch","articleBody":"## Expected Behavior \r\n\r\nDeploy feast v0.6-branch using docker-compose without errors (following https://docs.feast.dev/v/v0.6-branch/getting-started/deploying-feast/docker-compose).\r\n\r\n## Current Behavior\r\n\r\nErrors occur when running the cells of `feast/examples/basic/basic.ipynb`\r\n\r\n## Steps to reproduce\r\n\r\nCreate GCP project, service account, bucket.\r\n\r\n```shell\r\nPROJECT=feast-example-project\r\nSERVICE_ACCOUNT=feast-service-account\r\nBUCKET=feast-bucket-31072020\r\n\r\ngcloud projects create $PROJECT\r\ngcloud config set project $PROJECT\r\ngcloud iam service-accounts create $SERVICE_ACCOUNT\r\n\r\ngcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$SERVICE_ACCOUNT@$PROJECT.iam.gserviceaccount.com --role roles/editor\r\n\r\ngsutil mb gs://$BUCKET\r\n\r\ncd infra/docker-compose/gcp-service-accounts\r\ngcloud iam service-accounts keys create credentials.json --iam-account $SERVICE_ACCOUNT@$PROJECT.iam.gserviceaccount.com\r\n```\r\n\r\nCreate `.env` file with:\r\n\r\n```shell\r\n# General\r\nCOMPOSE_PROJECT_NAME=feast\r\nFEAST_VERSION=0.6.1\r\nFEAST_REPOSITORY_VERSION=v0.6-branch\r\n\r\n# Feast Core\r\nFEAST_CORE_IMAGE=gcr.io/kf-feast/feast-core\r\nFEAST_CORE_CONFIG=core.yml\r\nFEAST_CORE_GCP_SERVICE_ACCOUNT_KEY=credentials.json\r\n\r\n# Feast Serving\r\nFEAST_SERVING_IMAGE=gcr.io/kf-feast/feast-serving\r\n# Feast Serving - Batch (BigQuery)\r\nFEAST_BATCH_SERVING_CONFIG=batch-serving.yml\r\nFEAST_BATCH_SERVING_GCP_SERVICE_ACCOUNT_KEY=credentials.json\r\n# Feast Serving - Online (Redis)\r\nFEAST_ONLINE_SERVING_CONFIG=online-serving.yml\r\n\r\n# Jupyter\r\nFEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY=credentials.json\r\n```\r\n\r\nUpdate `batch-serving.yaml` to:\r\n\r\n```yaml\r\nfeast:\r\n  core-host: core\r\n  active-store: historical\r\n  stores:\r\n    - name: historical\r\n      type: BIGQUERY\r\n      # Changes required for batch serving to work\r\n      # Please see https://api.docs.feast.dev/grpc/feast.core.pb.html#Store for configuration options\r\n      config:\r\n        project_id: feast-example-project\r\n        dataset_id: dataset\r\n        staging_location: gs://feast-bucket-31072020/test\r\n        initial_retry_delay_seconds: 1\r\n        total_timeout_seconds: 21600\r\n      subscriptions:\r\n        - name: \"*\"\r\n          project: \"*\"\r\n  job_store:\r\n    redis_host: redis\r\n    redis_port: 6379\r\n  \r\ngrpc:\r\n  port: 6567\r\n```\r\n\r\nBring containers up using:\r\n\r\n```shell\r\ndocker container prune -f \u0026\u0026 docker volume prune -f \u0026\u0026 docker network prune -f\r\ndocker pull gcr.io/kf-feast/feast-serving:0.6.1\r\ndocker pull gcr.io/kf-feast/feast-core:0.6.1\r\ndocker-compose -f docker-compose.yml -f docker-compose.online.yml -f docker-compose.batch.yml up\r\n```\r\n\r\nA `java.lang.IllegalArgumentException` error is shown in the core logs upon running `client.apply(customer_fs)` in the notebook:\r\n\r\n```\r\n$ docker logs feast_core_1\r\nWARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.\r\n\r\n███████╗███████╗ █████╗ ███████╗████████╗\r\n██╔════╝██╔════╝██╔══██╗██╔════╝╚══██╔══╝\r\n█████╗  █████╗  ███████║███████╗   ██║   \r\n██╔══╝  ██╔══╝  ██╔══██║╚════██║   ██║   \r\n██║     ███████╗██║  ██║███████║   ██║   \r\n╚═╝     ╚══════╝╚═╝  ╚═╝╚══════╝   ╚═╝   \r\n                                         \r\n ██████╗ ██████╗ ██████╗ ███████╗        \r\n██╔════╝██╔═══██╗██╔══██╗██╔════╝        \r\n██║     ██║   ██║██████╔╝█████╗          \r\n██║     ██║   ██║██╔══██╗██╔══╝          \r\n╚██████╗╚██████╔╝██║  ██║███████╗        \r\n ╚═════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝        \r\n\r\n2020-07-31 14:06:35.558  INFO 12f08993e2f0 --- [           main] f.c.CoreApplication                      : Starting CoreApplication on 12f08993e2f0 with PID 1 (/opt/feast/feast-core.jar started by root in /)\r\n2020-07-31 14:06:35.579 DEBUG 12f08993e2f0 --- [           main] f.c.CoreApplication                      : Running with Spring Boot v2.0.9.RELEASE, Spring v5.0.13.RELEASE\r\n2020-07-31 14:06:35.583  INFO 12f08993e2f0 --- [           main] f.c.CoreApplication                      : No active profile set, falling back to default profiles: default\r\n2020-07-31 14:06:35.926  INFO 12f08993e2f0 --- [           main] ConfigServletWebServerApplicationContext : Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@7354b8c5: startup date [Fri Jul 31 14:06:35 UTC 2020]; root of context hierarchy\r\n2020-07-31 14:06:40.170  INFO 12f08993e2f0 --- [           main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring\r\n2020-07-31 14:06:40.657  INFO 12f08993e2f0 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.kafka.annotation.KafkaBootstrapConfiguration' of type [org.springframework.kafka.annotation.KafkaBootstrapConfiguration$$EnhancerBySpringCGLIB$$a4fe9a98] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)\r\n2020-07-31 14:06:40.764  INFO 12f08993e2f0 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$d14c8915] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)\r\n2020-07-31 14:06:41.667  INFO 12f08993e2f0 --- [           main] o.s.b.w.e.t.TomcatWebServer              : Tomcat initialized with port(s): 8080 (http)\r\n2020-07-31 14:06:41.708  INFO 12f08993e2f0 --- [           main] o.a.c.h.Http11NioProtocol                : Initializing ProtocolHandler [\"http-nio-8080\"]\r\n2020-07-31 14:06:41.749  INFO 12f08993e2f0 --- [           main] o.a.c.c.StandardService                  : Starting service [Tomcat]\r\n2020-07-31 14:06:41.756  INFO 12f08993e2f0 --- [           main] o.a.c.c.StandardEngine                   : Starting Servlet Engine: Apache Tomcat/8.5.39\r\n2020-07-31 14:06:41.778  INFO 12f08993e2f0 --- [ost-startStop-1] o.a.c.c.AprLifecycleListener             : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]\r\n2020-07-31 14:06:41.928  INFO 12f08993e2f0 --- [ost-startStop-1] o.a.c.c.C.[.[.[/]                        : Initializing Spring embedded WebApplicationContext\r\n2020-07-31 14:06:41.928  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.w.c.ContextLoader                    : Root WebApplicationContext: initialization completed in 6015 ms\r\n2020-07-31 14:06:43.014  WARN 12f08993e2f0 --- [ost-startStop-1] o.f.c.Flyway                             : Direct configuration of the Flyway object has been deprecated and will be removed in Flyway 6.0. Use Flyway.configure() instead.\r\n2020-07-31 14:06:43.015  WARN 12f08993e2f0 --- [ost-startStop-1] o.f.c.Flyway                             : Direct configuration retrieval from the Flyway object has been deprecated and will be removed in Flyway 6.0. Use Flyway.getConfiguration() instead.\r\n2020-07-31 14:06:43.016  WARN 12f08993e2f0 --- [ost-startStop-1] o.f.c.Flyway                             : Direct configuration retrieval from the Flyway object has been deprecated and will be removed in Flyway 6.0. Use Flyway.getConfiguration() instead.\r\n2020-07-31 14:06:43.016  WARN 12f08993e2f0 --- [ost-startStop-1] o.f.c.Flyway                             : Direct configuration of the Flyway object has been deprecated and will be removed in Flyway 6.0. Use Flyway.configure() instead.\r\n2020-07-31 14:06:43.045  WARN 12f08993e2f0 --- [ost-startStop-1] o.f.c.Flyway                             : Direct configuration of the Flyway object has been deprecated and will be removed in Flyway 6.0. Use Flyway.configure() instead.\r\n2020-07-31 14:06:43.076  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.l.VersionPrinter                 : Flyway Community Edition 5.2.4 by Boxfuse\r\n2020-07-31 14:06:43.107  INFO 12f08993e2f0 --- [ost-startStop-1] c.z.h.HikariDataSource                   : HikariPool-1 - Starting...\r\n2020-07-31 14:06:43.311  INFO 12f08993e2f0 --- [ost-startStop-1] c.z.h.HikariDataSource                   : HikariPool-1 - Start completed.\r\n2020-07-31 14:06:43.327  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.d.DatabaseFactory                : Database: jdbc:postgresql://db:5432/postgres (PostgreSQL 12.3)\r\n2020-07-31 14:06:43.341  WARN 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.d.b.Database                     : Flyway upgrade recommended: PostgreSQL 12.3 is newer than this version of Flyway and support has not been tested.\r\n2020-07-31 14:06:43.513  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbValidate                     : Successfully validated 7 migrations (execution time 00:00.080s)\r\n2020-07-31 14:06:43.529  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.s.JdbcTableSchemaHistory         : Creating Schema History table: \"public\".\"flyway_schema_history\"\r\n2020-07-31 14:06:43.563  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Current version of schema \"public\": \u003c\u003c Empty Schema \u003e\u003e\r\n2020-07-31 14:06:43.566  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema \"public\" to version 1 - Baseline\r\n2020-07-31 14:06:43.643  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema \"public\" to version 2 - RELEASE 0.6 Generalizing Source AND Extending FeatureSetJobStatus AND Feature Statistics\r\n2020-07-31 14:06:43.690  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema \"public\" to version 2.1 - Many Stores Per Job\r\n2020-07-31 14:06:43.713  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema \"public\" to version 2.2 - Subscription Migration\r\n2020-07-31 14:06:43.724  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema \"public\" to version 2.3 - Fix Primary Keys\r\n2020-07-31 14:06:43.740  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema \"public\" to version 2.4 - Store proto\r\n2020-07-31 14:06:43.751  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema \"public\" to version 2.5 - Fix Subscription MIgration\r\n2020-07-31 14:06:43.764  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Successfully applied 7 migrations to schema \"public\" (execution time 00:00.235s)\r\n2020-07-31 14:06:44.314  INFO 12f08993e2f0 --- [ost-startStop-1] j.LocalContainerEntityManagerFactoryBean : Building JPA container EntityManagerFactory for persistence unit 'default'\r\n2020-07-31 14:06:44.349  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.j.i.u.LogHelper                      : HHH000204: Processing PersistenceUnitInfo [\r\n\tname: default\r\n\t...]\r\n2020-07-31 14:06:44.508  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.Version                              : HHH000412: Hibernate Core {5.3.6.Final}\r\n2020-07-31 14:06:44.512  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.c.Environment                        : HHH000206: hibernate.properties not found\r\n2020-07-31 14:06:44.890  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.a.c.Version                          : HCANN000001: Hibernate Commons Annotations {5.0.4.Final}\r\n2020-07-31 14:06:45.711  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.d.Dialect                            : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL95Dialect\r\n2020-07-31 14:06:45.951  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.e.j.e.i.LobCreatorBuilderImpl        : HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException\r\n2020-07-31 14:06:45.967  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.t.BasicTypeRegistry                  : HHH000270: Type registration [java.util.UUID] overrides previous : org.hibernate.type.UUIDBinaryType@573f34e5\r\n2020-07-31 14:06:48.313  INFO 12f08993e2f0 --- [ost-startStop-1] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'\r\n2020-07-31 14:06:49.694  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.h.i.QueryTranslatorFactoryInitiator  : HHH000397: Using ASTQueryTranslatorFactory\r\n2020-07-31 14:06:51.060  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.FilterRegistrationBean         : Mapping filter: 'characterEncodingFilter' to: [/*]\r\n2020-07-31 14:06:51.060  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.FilterRegistrationBean         : Mapping filter: 'webMvcMetricsFilter' to: [/*]\r\n2020-07-31 14:06:51.061  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.FilterRegistrationBean         : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]\r\n2020-07-31 14:06:51.061  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.FilterRegistrationBean         : Mapping filter: 'httpPutFormContentFilter' to: [/*]\r\n2020-07-31 14:06:51.061  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.FilterRegistrationBean         : Mapping filter: 'requestContextFilter' to: [/*]\r\n2020-07-31 14:06:51.062  INFO 12f08993e2f0 --- [ost-startStop-1] .s.DelegatingFilterProxyRegistrationBean : Mapping filter: 'springSecurityFilterChain' to: [/*]\r\n2020-07-31 14:06:51.062  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.FilterRegistrationBean         : Mapping filter: 'httpTraceFilter' to: [/*]\r\n2020-07-31 14:06:51.063  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.ServletRegistrationBean        : Servlet metricsServlet mapped to [/metrics]\r\n2020-07-31 14:06:51.068  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.ServletRegistrationBean        : Servlet dispatcherServlet mapped to [/]\r\n2020-07-31 14:06:51.712  INFO 12f08993e2f0 --- [           main] .s.s.UserDetailsServiceAutoConfiguration : \r\n\r\nUsing generated security password: e2917407-3725-45d4-9e7a-e1772dcc4092\r\n\r\n2020-07-31 14:06:51.993  INFO 12f08993e2f0 --- [           main] o.s.s.w.DefaultSecurityFilterChain       : Creating filter chain: any request, [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@a565cbd, org.springframework.security.web.context.SecurityContextPersistenceFilter@481e91b6, org.springframework.security.web.header.HeaderWriterFilter@1daf3b44, org.springframework.security.web.authentication.logout.LogoutFilter@6f139fc9, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@3cbf1ba4, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@383864d5, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@6fc29daa, org.springframework.security.web.session.SessionManagementFilter@8a2a6a, org.springframework.security.web.access.ExceptionTranslationFilter@3ee69ad8, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@3cb8c8ce]\r\n2020-07-31 14:06:52.253  INFO 12f08993e2f0 --- [           main] o.s.w.s.h.SimpleUrlHandlerMapping        : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]\r\n2020-07-31 14:06:52.531  INFO 12f08993e2f0 --- [           main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@7354b8c5: startup date [Fri Jul 31 14:06:35 UTC 2020]; root of context hierarchy\r\n2020-07-31 14:06:52.616  WARN 12f08993e2f0 --- [           main] aWebConfiguration$JpaWebMvcConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning\r\n2020-07-31 14:06:52.680  INFO 12f08993e2f0 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped \"{[/error],produces=[text/html]}\" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)\r\n2020-07-31 14:06:52.681  INFO 12f08993e2f0 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped \"{[/error]}\" onto public org.springframework.http.ResponseEntity\u003cjava.util.Map\u003cjava.lang.String, java.lang.Object\u003e\u003e org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)\r\n2020-07-31 14:06:52.737  INFO 12f08993e2f0 --- [           main] o.s.w.s.h.SimpleUrlHandlerMapping        : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]\r\n2020-07-31 14:06:52.737  INFO 12f08993e2f0 --- [           main] o.s.w.s.h.SimpleUrlHandlerMapping        : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]\r\n2020-07-31 14:06:54.006  INFO 12f08993e2f0 --- [           main] o.s.b.a.e.w.EndpointLinksResolver        : Exposing 2 endpoint(s) beneath base path '/actuator'\r\n2020-07-31 14:06:54.039  INFO 12f08993e2f0 --- [           main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped \"{[/actuator/health],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}\" onto public java.lang.Object org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(javax.servlet.http.HttpServletRequest,java.util.Map\u003cjava.lang.String, java.lang.String\u003e)\r\n2020-07-31 14:06:54.040  INFO 12f08993e2f0 --- [           main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped \"{[/actuator/info],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}\" onto public java.lang.Object org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(javax.servlet.http.HttpServletRequest,java.util.Map\u003cjava.lang.String, java.lang.String\u003e)\r\n2020-07-31 14:06:54.042  INFO 12f08993e2f0 --- [           main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped \"{[/actuator],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}\" onto protected java.util.Map\u003cjava.lang.String, java.util.Map\u003cjava.lang.String, org.springframework.boot.actuate.endpoint.web.Link\u003e\u003e org.springframework.boot.actuate.endpoint.web.servlet.WebMvcEndpointHandlerMapping.links(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)\r\n2020-07-31 14:06:54.239  INFO 12f08993e2f0 --- [           main] o.a.k.c.a.AdminClientConfig              : AdminClientConfig values: \r\n\tbootstrap.servers = [kafka:9092, localhost:9094]\r\n\tclient.dns.lookup = default\r\n\tclient.id = \r\n\tconnections.max.idle.ms = 300000\r\n\tmetadata.max.age.ms = 300000\r\n\tmetric.reporters = []\r\n\tmetrics.num.samples = 2\r\n\tmetrics.recording.level = INFO\r\n\tmetrics.sample.window.ms = 30000\r\n\treceive.buffer.bytes = 65536\r\n\treconnect.backoff.max.ms = 1000\r\n\treconnect.backoff.ms = 50\r\n\trequest.timeout.ms = 15000\r\n\tretries = 5\r\n\tretry.backoff.ms = 100\r\n\tsasl.client.callback.handler.class = null\r\n\tsasl.jaas.config = null\r\n\tsasl.kerberos.kinit.cmd = /usr/bin/kinit\r\n\tsasl.kerberos.min.time.before.relogin = 60000\r\n\tsasl.kerberos.service.name = null\r\n\tsasl.kerberos.ticket.renew.jitter = 0.05\r\n\tsasl.kerberos.ticket.renew.window.factor = 0.8\r\n\tsasl.login.callback.handler.class = null\r\n\tsasl.login.class = null\r\n\tsasl.login.refresh.buffer.seconds = 300\r\n\tsasl.login.refresh.min.period.seconds = 60\r\n\tsasl.login.refresh.window.factor = 0.8\r\n\tsasl.login.refresh.window.jitter = 0.05\r\n\tsasl.mechanism = GSSAPI\r\n\tsecurity.protocol = PLAINTEXT\r\n\tsend.buffer.bytes = 131072\r\n\tssl.cipher.suites = null\r\n\tssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]\r\n\tssl.endpoint.identification.algorithm = https\r\n\tssl.key.password = null\r\n\tssl.keymanager.algorithm = SunX509\r\n\tssl.keystore.location = null\r\n\tssl.keystore.password = null\r\n\tssl.keystore.type = JKS\r\n\tssl.protocol = TLS\r\n\tssl.provider = null\r\n\tssl.secure.random.implementation = null\r\n\tssl.trustmanager.algorithm = PKIX\r\n\tssl.truststore.location = null\r\n\tssl.truststore.password = null\r\n\tssl.truststore.type = JKS\r\n\r\n2020-07-31 14:06:54.344  INFO 12f08993e2f0 --- [           main] o.a.k.c.u.AppInfoParser                  : Kafka version: 2.3.0\r\n2020-07-31 14:06:54.344  INFO 12f08993e2f0 --- [           main] o.a.k.c.u.AppInfoParser                  : Kafka commitId: fc1aaa116b661c8a\r\n2020-07-31 14:06:54.344  INFO 12f08993e2f0 --- [           main] o.a.k.c.u.AppInfoParser                  : Kafka startTimeMs: 1596204414340\r\n2020-07-31 14:06:54.892 ERROR 12f08993e2f0 --- [           main] o.s.k.c.KafkaAdmin                       : Failed to create topics\r\norg.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.\r\n2020-07-31 14:06:54.903  INFO 12f08993e2f0 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup\r\n2020-07-31 14:06:54.908  INFO 12f08993e2f0 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'dataSource' has been autodetected for JMX exposure\r\n2020-07-31 14:06:54.932  INFO 12f08993e2f0 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located MBean 'dataSource': registering with JMX server as MBean [com.zaxxer.hikari:name=dataSource,type=HikariDataSource]\r\n2020-07-31 14:06:55.019  INFO 12f08993e2f0 --- [           main] o.s.c.s.DefaultLifecycleProcessor        : Starting beans in phase 2147483547\r\n2020-07-31 14:06:55.049  INFO 12f08993e2f0 --- [           main] o.a.k.c.c.ConsumerConfig                 : ConsumerConfig values: \r\n\tallow.auto.create.topics = true\r\n\tauto.commit.interval.ms = 5000\r\n\tauto.offset.reset = latest\r\n\tbootstrap.servers = [kafka:9092, localhost:9094]\r\n\tcheck.crcs = true\r\n\tclient.dns.lookup = default\r\n\tclient.id = \r\n\tclient.rack = \r\n\tconnections.max.idle.ms = 540000\r\n\tdefault.api.timeout.ms = 60000\r\n\tenable.auto.commit = true\r\n\texclude.internal.topics = true\r\n\tfetch.max.bytes = 52428800\r\n\tfetch.max.wait.ms = 500\r\n\tfetch.min.bytes = 1\r\n\tgroup.id = core-service-null\r\n\tgroup.instance.id = null\r\n\theartbeat.interval.ms = 3000\r\n\tinterceptor.classes = []\r\n\tinternal.leave.group.on.close = true\r\n\tisolation.level = read_uncommitted\r\n\tkey.deserializer = class org.apache.kafka.common.serialization.StringDeserializer\r\n\tmax.partition.fetch.bytes = 1048576\r\n\tmax.poll.interval.ms = 300000\r\n\tmax.poll.records = 500\r\n\tmetadata.max.age.ms = 300000\r\n\tmetric.reporters = []\r\n\tmetrics.num.samples = 2\r\n\tmetrics.recording.level = INFO\r\n\tmetrics.sample.window.ms = 30000\r\n\tpartition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]\r\n\treceive.buffer.bytes = 65536\r\n\treconnect.backoff.max.ms = 1000\r\n\treconnect.backoff.ms = 50\r\n\trequest.timeout.ms = 30000\r\n\tretry.backoff.ms = 100\r\n\tsasl.client.callback.handler.class = null\r\n\tsasl.jaas.config = null\r\n\tsasl.kerberos.kinit.cmd = /usr/bin/kinit\r\n\tsasl.kerberos.min.time.before.relogin = 60000\r\n\tsasl.kerberos.service.name = null\r\n\tsasl.kerberos.ticket.renew.jitter = 0.05\r\n\tsasl.kerberos.ticket.renew.window.factor = 0.8\r\n\tsasl.login.callback.handler.class = null\r\n\tsasl.login.class = null\r\n\tsasl.login.refresh.buffer.seconds = 300\r\n\tsasl.login.refresh.min.period.seconds = 60\r\n\tsasl.login.refresh.window.factor = 0.8\r\n\tsasl.login.refresh.window.jitter = 0.05\r\n\tsasl.mechanism = GSSAPI\r\n\tsecurity.protocol = PLAINTEXT\r\n\tsend.buffer.bytes = 131072\r\n\tsession.timeout.ms = 10000\r\n\tssl.cipher.suites = null\r\n\tssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]\r\n\tssl.endpoint.identification.algorithm = https\r\n\tssl.key.password = null\r\n\tssl.keymanager.algorithm = SunX509\r\n\tssl.keystore.location = null\r\n\tssl.keystore.password = null\r\n\tssl.keystore.type = JKS\r\n\tssl.protocol = TLS\r\n\tssl.provider = null\r\n\tssl.secure.random.implementation = null\r\n\tssl.trustmanager.algorithm = PKIX\r\n\tssl.truststore.location = null\r\n\tssl.truststore.password = null\r\n\tssl.truststore.type = JKS\r\n\tvalue.deserializer = class feast.core.util.KafkaSerialization$ProtoDeserializer\r\n\r\n2020-07-31 14:06:55.119  INFO 12f08993e2f0 --- [           main] o.a.k.c.u.AppInfoParser                  : Kafka version: 2.3.0\r\n2020-07-31 14:06:55.119  INFO 12f08993e2f0 --- [           main] o.a.k.c.u.AppInfoParser                  : Kafka commitId: fc1aaa116b661c8a\r\n2020-07-31 14:06:55.119  INFO 12f08993e2f0 --- [           main] o.a.k.c.u.AppInfoParser                  : Kafka startTimeMs: 1596204415119\r\n2020-07-31 14:06:55.124  INFO 12f08993e2f0 --- [           main] o.a.k.c.c.KafkaConsumer                  : [Consumer clientId=consumer-1, groupId=core-service-null] Subscribed to topic(s): feast-specs-ack\r\n2020-07-31 14:06:55.130  INFO 12f08993e2f0 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService\r\n2020-07-31 14:06:55.139  INFO 12f08993e2f0 --- [           main] o.s.c.s.DefaultLifecycleProcessor        : Starting beans in phase 2147483647\r\n2020-07-31 14:06:55.151  WARN 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.NetworkClient                    : [Consumer clientId=consumer-1, groupId=core-service-null] Connection to node -2 (localhost/127.0.0.1:9094) could not be established. Broker may not be available.\r\n2020-07-31 14:06:55.175  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.Metadata                         : [Consumer clientId=consumer-1, groupId=core-service-null] Cluster ID: Rlqi6O23SFai-FI3_al8Sg\r\n2020-07-31 14:06:55.195  INFO 12f08993e2f0 --- [           main] n.d.b.g.s.s.AbstractGrpcServerFactory    : Registered gRPC service: feast.core.CoreService, bean: coreServiceImpl, class: feast.core.grpc.CoreServiceImpl\r\n2020-07-31 14:06:55.202  INFO 12f08993e2f0 --- [           main] n.d.b.g.s.s.AbstractGrpcServerFactory    : Registered gRPC service: grpc.health.v1.Health, bean: healthServiceImpl, class: feast.core.grpc.HealthServiceImpl\r\n2020-07-31 14:06:55.673  INFO 12f08993e2f0 --- [           main] n.d.b.g.s.s.GrpcServerLifecycle          : gRPC Server started, listening on address: *, port: 6565\r\n2020-07-31 14:06:55.713  INFO 12f08993e2f0 --- [           main] s.a.ScheduledAnnotationBeanPostProcessor : No TaskScheduler/ScheduledExecutorService bean found for scheduled processing\r\n2020-07-31 14:06:55.749  INFO 12f08993e2f0 --- [           main] o.a.c.h.Http11NioProtocol                : Starting ProtocolHandler [\"http-nio-8080\"]\r\n2020-07-31 14:06:55.751  INFO 12f08993e2f0 --- [pool-3-thread-1] f.c.s.JobCoordinatorService              : Polling for new jobs...\r\n2020-07-31 14:06:55.762  INFO 12f08993e2f0 --- [           main] o.a.t.u.n.NioSelectorPool                : Using a shared selector for servlet write/read\r\n2020-07-31 14:06:55.925  INFO 12f08993e2f0 --- [pool-3-thread-1] f.c.s.JobCoordinatorService              : No jobs found.\r\n2020-07-31 14:06:55.964  INFO 12f08993e2f0 --- [           main] o.s.b.w.e.t.TomcatWebServer              : Tomcat started on port(s): 8080 (http) with context path ''\r\n2020-07-31 14:06:55.969  INFO 12f08993e2f0 --- [           main] f.c.CoreApplication                      : Started CoreApplication in 21.761 seconds (JVM running for 24.3)\r\n2020-07-31 14:06:56.246  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.AbstractCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] Discovered group coordinator kafka:9092 (id: 2147482646 rack: null)\r\n2020-07-31 14:06:56.222  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.ConsumerCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] Revoking previously assigned partitions []\r\n2020-07-31 14:06:56.222  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: []\r\n2020-07-31 14:06:56.223  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.AbstractCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] (Re-)joining group\r\n2020-07-31 14:06:56.263  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.AbstractCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] (Re-)joining group\r\n2020-07-31 14:06:59.390  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.AbstractCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] Successfully joined group with generation 1\r\n2020-07-31 14:06:59.399  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.ConsumerCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] Setting newly assigned partitions: feast-specs-ack-0\r\n2020-07-31 14:06:59.424  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.ConsumerCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] Found no committed offset for partition feast-specs-ack-0\r\n2020-07-31 14:06:59.456  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.SubscriptionState            : [Consumer clientId=consumer-1, groupId=core-service-null] Resetting offset for partition feast-specs-ack-0 to offset 0.\r\n2020-07-31 14:06:59.461  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [feast-specs-ack-0]\r\n2020-07-31 14:07:25.894  INFO 12f08993e2f0 --- [pool-3-thread-1] f.c.s.JobCoordinatorService              : Polling for new jobs...\r\n2020-07-31 14:07:25.922  INFO 12f08993e2f0 --- [pool-3-thread-1] f.c.s.JobCoordinatorService              : Creating/Updating 2 jobs...\r\n2020-07-31 14:07:25.926 AUDIT 12f08993e2f0 --- [pool-4-thread-1] f.c.l.AuditLogger                        : {action=SUBMIT, detail=Building graph and submitting to DirectRunner, id=kafka-2111741832-to-historical-1596204445910, resource=JOB, timestamp=Fri Jul 31 14:07:25 UTC 2020}\r\n2020-07-31 14:07:25.926 AUDIT 12f08993e2f0 --- [pool-4-thread-2] f.c.l.AuditLogger                        : {action=SUBMIT, detail=Building graph and submitting to DirectRunner, id=kafka-2111741832-to-online-1596204445919, resource=JOB, timestamp=Fri Jul 31 14:07:25 UTC 2020}\r\n2020-07-31 14:07:26.866  INFO 12f08993e2f0 --- [pool-4-thread-2] f.i.ImportJob                            : Starting import job with settings: \r\nCurrent Settings:\r\n  appName: DirectRunnerJobManager\r\n  blockOnRun: false\r\n  defaultFeastProject: default\r\n  enforceEncodability: true\r\n  enforceImmutability: true\r\n  gcsPerformanceMetrics: false\r\n  jobName: kafka-2111741832-to-online-1596204445919\r\n  optionsId: 1\r\n  project: \r\n  runner: class org.apache.beam.runners.direct.DirectRunner\r\n  sourceJson: {\r\n  \"type\": \"KAFKA\",\r\n  \"kafkaSourceConfig\": {\r\n    \"bootstrapServers\": \"kafka:9092,localhost:9094\",\r\n    \"topic\": \"feast-features\"\r\n  }\r\n}\r\n  specsStreamingUpdateConfigJson: {\r\n  \"source\": {\r\n    \"bootstrapServers\": \"kafka:9092,localhost:9094\",\r\n    \"topic\": \"feast-specs\"\r\n  },\r\n  \"ack\": {\r\n    \"bootstrapServers\": \"kafka:9092,localhost:9094\",\r\n    \"topic\": \"feast-specs-ack\"\r\n  }\r\n}\r\n  stableUniqueNames: WARNING\r\n  storesJson: [{\r\n  \"name\": \"online\",\r\n  \"type\": \"REDIS\",\r\n  \"subscriptions\": [{\r\n    \"name\": \"*\",\r\n    \"project\": \"*\"\r\n  }],\r\n  \"redisConfig\": {\r\n    \"host\": \"redis\",\r\n    \"port\": 6379\r\n  }\r\n}]\r\n\r\n2020-07-31 14:07:26.866  INFO 12f08993e2f0 --- [pool-4-thread-1] f.i.ImportJob                            : Starting import job with settings: \r\nCurrent Settings:\r\n  appName: DirectRunnerJobManager\r\n  blockOnRun: false\r\n  defaultFeastProject: default\r\n  enforceEncodability: true\r\n  enforceImmutability: true\r\n  gcsPerformanceMetrics: false\r\n  jobName: kafka-2111741832-to-historical-1596204445910\r\n  optionsId: 0\r\n  project: \r\n  runner: class org.apache.beam.runners.direct.DirectRunner\r\n  sourceJson: {\r\n  \"type\": \"KAFKA\",\r\n  \"kafkaSourceConfig\": {\r\n    \"bootstrapServers\": \"kafka:9092,localhost:9094\",\r\n    \"topic\": \"feast-features\"\r\n  }\r\n}\r\n  specsStreamingUpdateConfigJson: {\r\n  \"source\": {\r\n    \"bootstrapServers\": \"kafka:9092,localhost:9094\",\r\n    \"topic\": \"feast-specs\"\r\n  },\r\n  \"ack\": {\r\n    \"bootstrapServers\": \"kafka:9092,localhost:9094\",\r\n    \"topic\": \"feast-specs-ack\"\r\n  }\r\n}\r\n  stableUniqueNames: WARNING\r\n  storesJson: [{\r\n  \"name\": \"historical\",\r\n  \"type\": \"BIGQUERY\",\r\n  \"subscriptions\": [{\r\n    \"name\": \"*\",\r\n    \"project\": \"*\"\r\n  }],\r\n  \"bigqueryConfig\": {\r\n    \"projectId\": \"feast-example-project\",\r\n    \"datasetId\": \"dataset\",\r\n    \"stagingLocation\": \"gs://feast-bucket-31072020/test\",\r\n    \"initialRetryDelaySeconds\": 1,\r\n    \"totalTimeoutSeconds\": 21600\r\n  }\r\n}]\r\n\r\n2020-07-31 14:07:27.685 ERROR 12f08993e2f0 --- [pool-4-thread-1] f.c.j.d.DirectRunnerJobManager           : Error submitting job\r\njava.lang.IllegalArgumentException: FixedWindows WindowingStrategies must have 0 \u003c= offset \u003c size\r\n\tat org.apache.beam.sdk.transforms.windowing.FixedWindows.\u003cinit\u003e(FixedWindows.java:65)\r\n\tat org.apache.beam.sdk.transforms.windowing.FixedWindows.of(FixedWindows.java:50)\r\n\tat feast.storage.connectors.bigquery.writer.BigQueryWrite.expand(BigQueryWrite.java:135)\r\n\tat feast.storage.connectors.bigquery.writer.BigQueryWrite.expand(BigQueryWrite.java:49)\r\n\tat org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539)\r\n\tat org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490)\r\n\tat org.apache.beam.sdk.values.PCollection.apply(PCollection.java:368)\r\n\tat feast.ingestion.ImportJob.runPipeline(ImportJob.java:167)\r\n\tat feast.core.job.direct.DirectRunnerJobManager.runPipeline(DirectRunnerJobManager.java:164)\r\n\tat feast.core.job.direct.DirectRunnerJobManager.startJob(DirectRunnerJobManager.java:83)\r\n\tat feast.core.job.CreateJobTask.call(CreateJobTask.java:48)\r\n\tat feast.core.job.CreateJobTask.call(CreateJobTask.java:29)\r\n\tat java.base/java.util.concurrent.FutureTask.run(Unknown Source)\r\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)\r\n\tat java.base/java.util.concurrent.FutureTask.run(Unknown Source)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\r\n\tat java.base/java.lang.Thread.run(Unknown Source)\r\n2020-07-31 14:07:27.686 ERROR 12f08993e2f0 --- [pool-4-thread-1] f.c.j.CreateJobTask                      : Error running ingestion job: java.lang.IllegalArgumentException: FixedWindows WindowingStrategies must have 0 \u003c= offset \u003c size\r\n2020-07-31 14:07:27.686 AUDIT 12f08993e2f0 --- [pool-4-thread-1] f.c.l.AuditLogger                        : {action=STATUS_CHANGE, detail=Job failed to be submitted to runner DirectRunner. Job status changed to ERROR., id=kafka-2111741832-to-historical-1596204445910, resource=JOB, timestamp=Fri Jul 31 14:07:27 UTC 2020}\r\n2020-07-31 14:07:27.939  INFO 12f08993e2f0 --- [pool-4-thread-2] i.l.c.EpollProvider                      : Starting without optional epoll library\r\n2020-07-31 14:07:27.943  INFO 12f08993e2f0 --- [pool-4-thread-2] i.l.c.KqueueProvider                     : Starting without optional kqueue library\r\n2020-07-31 14:07:28.971  INFO 12f08993e2f0 --- [pool-4-thread-2] o.a.k.c.c.ConsumerConfig                 : ConsumerConfig values: \r\n```\r\n\r\n\r\n### Specifications\r\n\r\ndocker desktop v2.2.0.4 on macOS 10.15.5\r\n\r\n## Possible Solution\r\n\r\nInclude `write_triggering_frequency_seconds` in `batch-serving.yaml`:\r\n\r\n```yaml\r\nfeast:\r\n  core-host: core\r\n  active-store: historical\r\n  stores:\r\n    - name: historical\r\n      type: BIGQUERY\r\n      # Changes required for batch serving to work\r\n      # Please see https://api.docs.feast.dev/grpc/feast.core.pb.html#Store for configuration options\r\n      config:\r\n        project_id: feast-example-project\r\n        dataset_id: dataset\r\n        staging_location: gs://feast-bucket-31072020/test\r\n        initial_retry_delay_seconds: 1\r\n        total_timeout_seconds: 21600\r\n        write_triggering_frequency_seconds: 1\r\n      subscriptions:\r\n        - name: \"*\"\r\n          project: \"*\"\r\n  job_store:\r\n    redis_host: redis\r\n    redis_port: 6379\r\n  \r\ngrpc:\r\n  port: 6567\r\n```\r\n\r\nAdding the trigger frequency setting changes the error to:\r\n\r\n```\r\n$ docker logs feast_core_1\r\n\r\n...\r\n\r\n2020-07-31 14:15:13.157 ERROR d70f6bada002 --- [pool-4-thread-2] f.c.j.d.DirectRunnerJobManager           : Error submitting job\r\njava.util.ConcurrentModificationException\r\n\tat java.base/java.util.HashMap.computeIfAbsent(Unknown Source)\r\n\tat org.apache.beam.sdk.schemas.utils.ReflectUtils.getMethods(ReflectUtils.java:77)\r\n\tat org.apache.beam.sdk.schemas.AutoValueSchema$AbstractGetterTypeSupplier.get(AutoValueSchema.java:46)\r\n\tat org.apache.beam.sdk.schemas.utils.StaticSchemaInference.schemaFromClass(StaticSchemaInference.java:83)\r\n\tat org.apache.beam.sdk.schemas.utils.JavaBeanUtils.schemaFromJavaBeanClass(JavaBeanUtils.java:63)\r\n\tat org.apache.beam.sdk.schemas.AutoValueSchema.schemaFor(AutoValueSchema.java:114)\r\n\tat org.apache.beam.sdk.schemas.annotations.DefaultSchema$DefaultSchemaProvider.schemaFor(DefaultSchema.java:131)\r\n\tat org.apache.beam.sdk.schemas.SchemaRegistry.lambda$getSchema$0(SchemaRegistry.java:256)\r\n\tat org.apache.beam.sdk.schemas.SchemaRegistry.getProviderResult(SchemaRegistry.java:239)\r\n\tat org.apache.beam.sdk.schemas.SchemaRegistry.getSchema(SchemaRegistry.java:256)\r\n\tat org.apache.beam.sdk.values.PCollection.inferCoderOrFail(PCollection.java:159)\r\n\tat org.apache.beam.sdk.values.PCollection.finishSpecifyingOutput(PCollection.java:94)\r\n\tat org.apache.beam.sdk.runners.TransformHierarchy.setOutput(TransformHierarchy.java:210)\r\n\tat org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:540)\r\n\tat org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490)\r\n\tat org.apache.beam.sdk.values.PCollection.apply(PCollection.java:368)\r\n\tat feast.ingestion.transform.ReadFromSource.expand(ReadFromSource.java:91)\r\n\tat feast.ingestion.transform.ReadFromSource.expand(ReadFromSource.java:37)\r\n\tat org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539)\r\n\tat org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490)\r\n\tat org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)\r\n\tat org.apache.beam.sdk.Pipeline.apply(Pipeline.java:184)\r\n\tat feast.ingestion.ImportJob.runPipeline(ImportJob.java:117)\r\n\tat feast.core.job.direct.DirectRunnerJobManager.runPipeline(DirectRunnerJobManager.java:164)\r\n\tat feast.core.job.direct.DirectRunnerJobManager.startJob(DirectRunnerJobManager.java:83)\r\n\tat feast.core.job.CreateJobTask.call(CreateJobTask.java:48)\r\n\tat feast.core.job.CreateJobTask.call(CreateJobTask.java:29)\r\n\tat java.base/java.util.concurrent.FutureTask.run(Unknown Source)\r\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)\r\n\tat java.base/java.util.concurrent.FutureTask.run(Unknown Source)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\r\n\tat java.base/java.lang.Thread.run(Unknown Source)\r\n2020-07-31 14:15:13.157 ERROR d70f6bada002 --- [pool-4-thread-2] f.c.j.CreateJobTask                      : Error running ingestion job: java.util.ConcurrentModificationException\r\n```\r\n","author":{"url":"https://github.com/masonlr","@type":"Person","name":"masonlr"},"datePublished":"2020-07-31T14:21:22.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/911/feast/issues/911"}