图计算
最大连通图ConnectedComponents会将输入的图作为无向图处理。
示例:
val spark = SparkSession .builder() .appName("k6k4Test") .master("local[4]") .getOrCreate() val sqlContext = spark.sqlContext // 创建顶点DataFrame val v = sqlContext.createDataFrame(List( ("a", "Alice", 34), ("b", "Bob", 36), ("c", "Charlie", 30), ("d", "David", 29), ("e", "Esther", 32), ("f", "Fanny", 36), ("g", "Alice", 60) )).toDF("id", "name", "age") // 创建边DataFrame val e = sqlContext.createDataFrame(List( ("a", "b", "friend"), ("b", "c", "follow"), ("c", "b", "follow"), ("f", "c", "follow"), ("e", "f", "follow"), ("e", "d", "friend"), ("d", "a", "friend"), ("a", "e", "friend") )).toDF("src", "dst", "relationship") val g = GraphFrame(v, e) val result = g.connectedComponents.setCheckpointInterval(0).run(); result.printSchema() result.show(false) //结果写入hive表 //result.write.saveAsTable("hiveTable") spark.stop()
结果输出:
|-- id: string (nullable = true)
|-- name: string (nullable = true)
|-- age: integer (nullable = false)
|-- component: long (nullable = true)
+---+-------+---+------------+
|id |name |age|component |
+---+-------+---+------------+
|a |Alice |34 |412316860416|
|b |Bob |36 |412316860416|
|c |Charlie|30 |412316860416|
|d |David |29 |412316860416|
|e |Esther |32 |412316860416|
|f |Fanny |36 |412316860416|
|g |Alice |60 |146028888064|
+---+-------+---+------------+