Spark

Spark笔记(九九)-Spark开发常见故障

By timebusker on January 25, 2020

Spark IDE: System memory 251396096 must be at least 471859200

spark运行内存不够导致
conf.set("spark.testing.memory", "1G")

SparkSQL-Hive通过scala API建表异常 For direct MetaStore DB connections, we don't support retries at the client level.

Hive元数据编码异常导致
hive元数据库编码默认要采用 latin1 
若中文乱码再单独解决特定表，不可全部改变编码格式为UTF-8

posgresql:
CREATE DATABASE maggieprod WITH ENCODING = 'LATIN1' LC_CTYPE = 'en_CA' LC_COLLATE = 'en_CA' TEMPLATE template0;

解决以上问题的办法是修改hive元数据TBLS表结构，设计为IS_REWRITE_ENABLED可为空或者设置默认值

ALTER TABLE "public"."TBLS" ALTER COLUMN "IS_REWRITE_ENABLED" DROP NOT NULL;

ALTER TABLE "public"."TBLS" ALTER COLUMN "IS_REWRITE_ENABLED" set default(true);
目录
    FEATURED TAGS
    CSS JAVA JQuery JQGrid Linux Java并发编程实战 开发工具 JAVA设计模式 ZooKeeper专题 Nginx Hive SparkSQL MyBatis Scala 博客专栏 消息中间件 Kafka Spring相关 Hadoop Flume 实践记录 KVM PostgreSQL Oracle Thinking-In-Java JAVA杂记 MySQL SQL Effective-Java Shell Sqoop Spark MongoDB SpringBoot RequireJS HBase Node.js Vue.js JVM调优 Greenplum Spark-Streaming 机器学习基础 读书笔记 Docker Spark-ML Maven 高数知识整理 技术知识汇总 FastDFS Redis 软考中级
    FRIENDS