A lightweight and ready-to-use Apache Spark environment for Windows development
1. spark3.3
2. hadoop3
3. Java11
4. Python3.7
5. maven3.6
6. VSCode1.72
1. Download all the files in this repository and copy the root folder[Apache] to C:\.
2. Execute the environment variable setup bat [C:\Apache\Spark3.3\tools\environment_variable_setup.bat].
1. In CMD line, you can run your sparkSQL file like :
[python C:\Apache\Spark3.3\tools\spark-3.3.0-bin-hadoop3\bin\spark-submit.py C:\Apache\Spark3.3\source\example.py]
2. You can also develop and run the script file in VScode's debug mode [SparkSubmit]
The setted VScode : [C:\Apache\Spark3.3\tools\VSCode-win32-x64-1.72.0\Code.exe]
1. Fork it (https://github.com/vekee/D-Tools.git)
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request
DUAN DAHAI