Jump to: navigation, search

Difference between revisions of "Image handling in edge environment"

Line 87: Line 87:
 
<br/>
 
<br/>
 
[[File:Glance-edge-architectures-api-servers.PNG]] <br/>
 
[[File:Glance-edge-architectures-api-servers.PNG]] <br/>
 +
A description in [https://www.dropbox.com/s/lr8ysy1j8uvpy6z/GlanceEdgeArchitecture-DC-StarlingX.pptx?dl=0 slides].
 
Cascading is not possible as only pulling strategy works in this case.
 
Cascading is not possible as only pulling strategy works in this case.
 
* Concerns/Questions
 
* Concerns/Questions
Line 96: Line 97:
 
** First usage of an image always takes a long time
 
** First usage of an image always takes a long time
 
** In case of network connection error to the central Glance Nova will have access to the images, but will not be able to figure out if the user have rights to use the image and will not have path to the images data
 
** In case of network connection error to the central Glance Nova will have access to the images, but will not be able to figure out if the user have rights to use the image and will not have path to the images data
 +
** At the moment Glance API cache does not support metadata caching.

Revision as of 11:19, 15 October 2018

This page contains a summary of the Vancouver Forum discussions about the topic. Full notes of the discussion are in here. The features and requirements for edge cloud infrastructure are described in OpenStack_Edge_Discussions_Dublin_PTG.
Source of the figures is here.

Synchronisation strategies

  • Copy every image to every edge cloud instance: the simplest and less optimal solution
  • Copy images only to those edge cloud instances where they are needed
    • Provide a syncronisation policy together with the image
    • Rely on the pulling of the images
  • Per clarkb, nodepool might be a good option here. It will aggressively try to ensure that the images you want are in all of the locations you want them in.

Architecture options for Glance

Legend for the figures:

Vancouver image edge notes figures xLegend.png

One Glance with multiple backends - pull mode

There is one central Glance what is capable to handle multiple backends. Every edge cloud instance is represented by a Glance backend in the central Glance. Every edge cloud instance runs a Glance API server which is configured in a way, that it uses the central Glance to pull the images, but also uses a local cache. All Glance API servers use a shared database and the database clustering is out of Glance-s reponsibility. Each Glance API server can access each others images, but use its own by default. When accessing a remote image Glance API streams the image from the remote location. (In this sense this is pull mode). OpenStack services in the edge cloud instances are using the images in the Glance backend with a direct url.
Vancouver image edge notes figures xOneGlanceWithMultipleBackends pull.png
Work for multiple backends has been already started with an etherpad and a spec. Cascading (one edge cloud instance is the receiver and the source of the images at the same time) is possible if the central Glance is able to orchestrate the syncronisation of images.

  • Concerns/Questions
    • Network partitioning tolerance?
      • Network connection is required when a given image is started for the first time.
    • Is it safe to store database credentials in the far edge? (It is not possible to provide image access without network connection and not store the database credentials in the edge cloud instance)
      • It is unavoidable in this case.
    • Can the OpenStack services running in the edge cloud instances use the images from the local Glance? There is a worry that OpenStack services (e.g. nova) still need to get the direct URL via the Glance API which is ONLY available at the central site.
      • This can be avoided by allowing Glance to access all the CEPH instances as backends
    • Is CEPH backend CEPH block of CEPH RGW in the figure?
      • Glance talks directly to the CEPH block. As an alternative it is possible to use Swift from Glance and use CEPH RGW as a Swift backend.
    • Did I got it correctly, that the CEPH backends are in a replication configuration?
      • Not always
  • Pros
    • Relatively easy to implement based on the current Glance architecture
    • Smaller storage needs on the edge cloud instances
    • Images are automatically loaded to the edge cloud instances from the central Galnce
  • Cons
    • Requires the same Glance backend in every edge cloud instance
    • Requires the same OpenStack version in every edge cloud instance (apart from during upgrade)
    • Sensitivity for network connection loss is not clear
    • There is no explicit control over the time period while an image is cached
    • It is not possible to invalidate an image from the cache
    • It is mandatory to store the Glance database credentials in every edge cloud instance
    • The images are pulled

One Glance with multiple backends - distributed storage

There is one central Glance what is capable to handle multiple backends. Every edge cloud instance is represented by a Glance backend in the central Glance. Every edge cloud instance runs a Glance API server which is configured in a way, that it uses the central Glance to pull the images, but also uses a local cache. All Glance API servers use a shared database and the database clustering is out of Glance-s reponsibility. The storage backend supports clustering and available in every edge cloud instance. The clustering of the storage is not in the scope of Glance OpenStack services in the edge cloud instances are using the images in the Glance backend with a direct url.
Vancouver image edge notes figures xOneGlanceWithMultipleBackends distributed storage.png
Work for multiple backends has been already started with an etherpad and a spec. In Rocky several backends are already supported, but they have to be different type. Support for same type multiple backends is not implemented yet. Cascading (one edge cloud instance is the receiver and the source of the images at the same time) is possible if the central Glance is able to orchestrate the syncronisation of images.

  • Concerns/Questions
  • Pros
    • Relatively easy to implement based on the current Glance architecture
  • Cons
    • Requires the same Glance backend in every edge cloud instance
    • Requires the same OpenStack version in every edge cloud instance (apart from during upgrade)
    • Sensitivity for network connection loss is not clear
    • Every image is replicated to every edge cloud instance
    • It is mandatory to store the Glance database credentials in every edge cloud instance
    • Adding new edged cloud instances require the reconfiguration of the distributed database

Several Glances with an independent syncronisation service, sych via Glance API

Every edge cloud instance have its Glance instance. There is a synchronisation service what is able to instruct the Glances to do the synchronisation. The synchronisation of the image data is done using Glance API-s.
Glance-edge-architecture-synch-service.PNG
Cascading is visible for the syncronisation service only, not for Glance.

  • Pros
    • Every edge cloud instance can have a different Glance backend
    • Can support multiple OpenStack versions in the different edge cloud instances
    • Can be extended to support multiple VIM types
    • Using the API provides support for schema change in the metadata
  • Cons
    • Needs a new synchronisation service
    • At this moment certain metadata is not visible for the API users and it is not possible to assign metadata to an image

Several Glances with an independent syncronisation service, synch using the backend

Every edge cloud instance have its Glance instance. There is a synchronisation service what is able to instruct the Glances to do the synchronisation. The synchronisation of the image data is the responsibility of the backend (eg.: CEPH). Vancouver image edge notes figures xSeveralGlancesWithAnIndependentSynchronisationServiceBackendSynch.png Cascading is visible for the syncronisation service only, not for Glance.

  • Concerns/Questions
    • Is CEPH backend CEPH block or CEPH RBD in the figure?
      • CEPH RBD
  • Pros
  • Cons
    • Needs a new synchronisation service

One Glance and multiple Glance API servers

There is one central Glance and every edge cloud instance runs a separate Glance API server. These Glance API servers are communicating with the central Glance. Backend is accessed centrally, there is only caching in the edge cloud instances. Nova Image Caching caches at the compute node what would work ok for all-in-one edge clouds or small edge clouds. But glance-api caching caches at the edge cloud level, so works better for large edge clouds with lots of compute nodes.
Glance-edge-architectures-api-servers.PNG
A description in slides. Cascading is not possible as only pulling strategy works in this case.

  • Concerns/Questions
    • Do we plan to use Nova image caching or caching in the Glance API server?
    • Are the image metadata also cached in Glance API server or only the images?
  • Pros
    • Implicitly location aware
  • Cons
    • First usage of an image always takes a long time
    • In case of network connection error to the central Glance Nova will have access to the images, but will not be able to figure out if the user have rights to use the image and will not have path to the images data
    • At the moment Glance API cache does not support metadata caching.