在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称:ms-operator开源软件地址:https://gitee.com/mindspore/ms-operator开源软件介绍:MindSpore OperatorExperimental notice: This project is still experimental and only serves as a proof of concept for running MindSpore on Kubernetes. The current version of ms-operator is based on an early version of PyTorch Operator and TF Operator. Right now MindSpore supports running LeNet with MNIST dataset on a single node, while distributed training examples are conducted using Volcano.Introduction of MindSpore and ms-operatorMindSpore is a new open source deep learning training/inference framework thatcould be used for mobile, edge and cloud scenarios. MindSpore is designed toprovide development experience with friendly design and efficient execution forthe data scientists and algorithmic engineers, native support for Ascend AIprocessor, and software hardware co-optimization. This project contains the specification and implementation of MSJob customresource definition. We will demonstrate running a walkthrough of creatingms-operator, as well as MNIST training job on Kubernetes with MindSpore-cpu image(x86 CPU build version) on a single node. More completed features will bedeveloped in the coming days. This project defines the following:
MindSpore docker imagePlease refer to MindSpore docker image introductionfor details. DesignThe yaml file we used to create our MNIST training job is defined as follows: apiVersion: v1kind: Podmetadata: name: msjob-mnistspec: containers: - image: mindspore/mindspore-cpu:0.1.0-alpha imagePullPolicy: IfNotPresent name: msjob-mnist command: ["/bin/bash", "-c", "python /tmp/test/MNIST/lenet.py"] volumeMounts: - name: training-result mountPath: /tmp/result - name: ms-mnist mountPath: /tmp/test restartPolicy: OnFailure volumes: - name: training-result emptyDir: {} - name: ms-mnist hostPath: path: /root/gopath/src/gitee.com/mindspore/ms-operator/examples/ Overview of MindSpore in Kubeflow ecosystemThe high-level view of how MindSpore fits in the ecosystem of Kubeflow and itscomponents. MindSpore CPU examplePrerequisites
StepsFirst, pull the ms-operator image from Docker Hub: docker pull mindspore/ms-operator:latest Or you can build the ms-operator image on local machine: go build -ldflags '-w -s' -o ms-operator cmd/ms-operator.v1/main.godocker build -t mindspore/ms-operator . After the installation, check the image status using REPOSITORY TAG IMAGE ID CREATED SIZEmindspore/ms-operator latest 4a17028de3d3 5 minutes ago 97.8MB The MindSpore image we download from docker hub is REPOSITORY TAG IMAGE ID CREATED SIZEmindspore/mindspore-cpu 0.1.0-alpha ef443be923bc 3 hours ago 1.05GB MindSpore supports heterogeneous computing including multiple hardware andbackends ( Install the msjob crd, ms-operator deployment and pod: RBAC=true # set false if you do not have an RBAC clusterhelm install ms-operator-chart/ -n ms-operator --set rbac.install=${RBAC} --wait --replace Using LAST DEPLOYED: Tue Mar 24 11:36:51 2020NAMESPACE: defaultSTATUS: DEPLOYEDRESOURCES:==> v1beta1/CustomResourceDefinitionNAME AGEmsjobs.kubeflow.org 1d==> v1beta1/DeploymentNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGEms-operator 1 1 1 1 1d==> v1/Pod(related)NAME READY STATUS RESTARTS AGEms-operator-7b5b457d69-dpd2b 1/1 Running 0 1d We will do a MNIST training to check the eligibility of MindSpore running onKubernetes: cd examples/ && kubectl apply -f ms-mnist.yaml The job is simply importing MindSpore packages, the dataset is already included in the kubectl get pod msjob-mnist && kubectl logs msjob-mnist The expected output is: NAME READY STATUS RESTARTS AGEmsjob-mnist 0/1 Completed 0 3h53m============== Starting Training ==============epoch: 1 step: 1, loss is 2.3005836epoch: 1 step: 2, loss is 2.2978227epoch: 1 step: 3, loss is 2.3004227epoch: 1 step: 4, loss is 2.3054247epoch: 1 step: 5, loss is 2.3068798epoch: 1 step: 6, loss is 2.298408epoch: 1 step: 7, loss is 2.3055573epoch: 1 step: 8, loss is 2.2998955epoch: 1 step: 9, loss is 2.3028255epoch: 1 step: 10, loss is 2.2972553 Distributed GPU example using VolcanoThe source code of the example can be found here. Volcano Prerequisites
Install Volcano: MindSpore GPU exampleUsing a modified image which the openssh-server is installed from the official MindSpore GPU image. To check the eligibility of MindSpore GPU's ability to communicate with other processes, we leverage the mpimaster and mpiworker task spec of Volcano. In this example, we launch one mpimaster and two mpiworkers, the python script is taken from MindSpore Gitee README, which is also modified to be able to run parallelly. cd to The expected output should be (2*3) of multi-dimensional array. Future WorkKubeflow just announced its first major1.0 release recently with the graduation of a core set of stable applicationsincluding:
The MindSpore community is driving to collaborate with the Kubeflow communityas well as making the ms-operator more complex, well-organized and itsdependencies up-to-date. All these components make it easy for machine learningengineers and data scientists to leverage cloud assets (public or on-premise)for machine learning workloads. MindSpore is also looking forward to enable users to use Jupyter to developmodels. Users in the future can use Kubeflow tools like fairing (Kubeflow’spython SDK) to build containers and create Kubernetes resources to train theirMindSpore models. Once training completed, users can use KFServingto create and deploy a server for inference thus completing the life cycle ofmachine learning. Community
ContributingWelcome contributions. See our Contributor Wiki for more details. License |
请发表评论