学习Spark

ForewordPreface1. Introduction to Data Analysis with SparkWhat Is Apache Spark?A Unified StackSpark CoreSpark SQLSpark StreamingMLlibGraphXCluster ManagersWho Uses Spark, and for What?Data Science TasksData Processing ApplicationsA Brief History of SparkSpark Versions and ReleasesStorage Layers for Spark2. Downloading Spark and Getting StartedDownloading SparkIntroduction to Spark's Python and Scala ShellsIntroduction to Core Spark ConceptsStandalone ApplicationsInitializing a SparkContextBuilding Standalone ApplicationsConclusion3. Programming with RDDsRDD BasicsCreating RDDsRDD OperationsTransformationsActionsLazy EvaluationPassing Functions to SparkPythonScalaJavaCommon Transformations and ActionsBasic RDDsConverting Between RDD TypesPersistence (Caching)Conclusion4. Working with Key/Value PairsMotivationCreating Pair RDDsTransformations on Pair RDDsAggregationsGrouping DataJoinsSorting DataActions Available on Pair RDDsData Partitioning (Advanced)Determining an RDD's PartitionerOperations That Benefit from PartitioningOperations That Affect PartitioningExample: PageRankCustom PartitionersConclusion5. Loading and Saving Your DataMotivationFile FormatsText FilesJSONComma-Separated Values and Tab-Separated ValuesSequenceFilesObject FilesHadoop Input and Output FormatsFile CompressionFilesystemsLocal/"Regular" FSAmazon $3HDFSStructured Data with Spark SQLApache HiveJSONDatabasesJava Database ConnectivityCassandraHBaseElasticsearchConclusion6. Advanced Spark ProgrammingIntroductionAccumulatorsAccumulators and Fault ToleranceCustom AccumulatorsBroadcast VariablesOptimizing BroadcastsWorking on a Per-Partition BasisPiping to External ProgramsNumeric RDD OperationsConclusion7. Running on a ClusterIntroductionSpark Runtime ArchitectureThe DriverExecutorsCluster ManagerLaunching a ProgramSummaryDeploying Applications with spark-submitPackaging Your Code and DependenciesA Java Spark Application Built with MavenA Scala Spark Application Built with sbtDependency ConflictsScheduling Within and Between Spark ApplicationsCluster ManagersStandalone Cluster ManagerHadoop YARNApache MesosAmazon EC2Which Cluster Manager to Use?Conclusion8. Tuning and Debugging SparkConfiguring Spark with SparkConfComponents of Execution: Jobs, Tasks, and StagesFinding InformationSpark Web UIDriver and Executor LogsKey Performance ConsiderationsLevel of ParallelismSerialization FormatMemory ManagementHardware ProvisioningConclusion9. Spark SQLLinking with Spark SQLUsing Spark SQL in ApplicationsInitializing Spark SQLBasic Query ExampleSchemaRDDsCachingLoading and Saving DataApache HiveParquetJSONFrom RDDsJDBC/ODBC ServerWorking with BeelineLong-Lived Tables and QueriesUser-Defined FunctionsSpark SQL UDFsHive UDFsSpark SQL PerformancePerformance Tuning OptionsConclusion10. Spark StreamingA Simple ExampleArchitecture and AbstractionTransformationsStateless TransformationsStateful TransformationsOutput OperationsInput SourcesCore SourcesAdditional SourcesMultiple Sources and Cluster Sizing24/7 OperationCheckpointingDriver Fault ToleranceWorker Fault ToleranceReceiver Fault ToleranceProcessing GuaranteesStreaming UIPerformance ConsiderationsBatch and Window SizesLevel of ParallelismGarbage Collection and Memory UsageConclusion11. Machine Learning with MLlibOverviewSystem RequirementsMachine Learning BasicsExample: Spam ClassificationData TypesWorking with VectorsAlgorithmsFeature ExtractionStatisticsClassification and RegressionClusteringCollaborative Filtering and RecommendationDimensionality ReductionModel EvaluationTips and Performance ConsiderationsPreparing FeaturesConfiguring AlgorithmsCaching RDDs to ReuseRecognizing SparsityLevel of ParallelismPipeline APIConclusionIndex