Spark

Spark笔记(九九)-Spark开发常见故障

By timebusker on January 25, 2020

Spark IDE: System memory 251396096 must be at least 471859200

spark运行内存不够导致
conf.set("spark.testing.memory", "1G")

SparkSQL-Hive通过scala API建表异常 For direct MetaStore DB connections, we don't support retries at the client level.

Hive元数据编码异常导致
hive元数据库编码默认要采用 latin1 
若中文乱码再单独解决特定表，不可全部改变编码格式为UTF-8

posgresql:
CREATE DATABASE maggieprod WITH ENCODING = 'LATIN1' LC_CTYPE = 'en_CA' LC_COLLATE = 'en_CA' TEMPLATE template0;

解决以上问题的办法是修改hive元数据TBLS表结构，设计为IS_REWRITE_ENABLED可为空或者设置默认值

ALTER TABLE "public"."TBLS" ALTER COLUMN "IS_REWRITE_ENABLED" DROP NOT NULL;

ALTER TABLE "public"."TBLS" ALTER COLUMN "IS_REWRITE_ENABLED" set default(true);
目录