source_manager.h File Reference

Source manager is what that read the source files and own the buffer that contains the data. More...

#include <stddef.h>
#include "serene/rt/fiber/mutex.h"
#include "serene/rt/fiber/thread.h"
#include "serene/rt/impl/array_list.h"
#include "serene/rt/impl/hashmap.h"
#include "serene/utils.h"

Include dependency graph for source_manager.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Data Structures
struct	srn_sm_maybe_id_t
struct	srn_sm_buffer_t
struct	srn_sm_maybe_buffer_t
struct	srn_sm_t

Macros
#define	SRN_SM_NO_ID   UINTPTR_MAX
#define	SM_TRACEPOINT(...)

Typedefs
typedef size_t	srn_sm_buffer_id_t
typedef struct srn_sm_maybe_id_t	srn_sm_maybe_id_t
typedef struct srn_sm_buffer_t	srn_sm_buffer_t
typedef struct srn_sm_maybe_buffer_t	srn_sm_maybe_buffer_t
typedef struct srn_sm_t	srn_sm_t

Functions
srn_sm_t *	srn_sm_init (srn_context_t *ctx)
	Initialize the source manager, as an immortal data structure.
void	srn_sm_shutdown (srn_sm_t *sm)
	Shutdown the source manager freeing all the buffers.
srn_sm_buffer_t *	srn_sm_get_buffer (srn_sm_t *sm, srn_sm_buffer_id_t id)
	Return the buffer pointer of the given id, nullptr if the id does not exists.
srn_sm_maybe_id_t	srn_sm_read_buffer_file (srn_sm_t *sm, const char *filepath, bool reread)
	Read and buffer the given filepath into the source manager and return the buffer id.
srn_sm_maybe_buffer_t	srn_sm_buffer_file (srn_sm_t *sm, const char *filepath, bool reread)
	Read and buffer the given filepath into the source manager and return a srn_sm_maybe_buffer_t which either fails or has a pointer to the buffer containing the content of the file.
static const unsigned char *	srn_sm_buffer_data (srn_sm_buffer_t *buf)
static size_t	srn_sm_buffer_len (srn_sm_buffer_t *buf)

Detailed Description

Source manager is what that read the source files and own the buffer that contains the data.

Notes:

• We use manual memory management for buffers since the size of buffers are not known in advance and they might be bigger than block sizes, also the lifecycle of buffers are known and we can avoidarena allocation completely.
• Each buffer is assigned a stable integer ID on first load. That ID is the canonical handle for all later references and never changes for a given file path.
• A hashmap from file path to ID prevents duplicate reads. Passing an already loaded path to srn_sm_read_buffer_file returns the existing ID without reopening the file.
• When reread is true, the freshly read buffer is prepended to the existing buffer's chain rather than replacing it. The slot ID stays the same. The previous buffer remains reachable via buffer->next this way expressions that are previously loaded will continue to have valid locations.
• An empty file produces a valid buffer slot with data == nullptr and len == 0. Callers must treat a null data field as a valid zero length source rather than an error.
• File content is read through srn_fiber_read, so the call suspends the current fiber instead of blocking the OS thread.
• A single write lock serialises all insertions and updates. A new file is read outside the lock and the map is checked again under it before the slot is inserted, so two fibers loading the same path end up sharing one slot. The path lookup snapshots the map under the lock; srn_sm_get_buffer is unsynchronised, callers must not call it concurrently with a write.
• The source manager itself is allocated in the immortal pool and lives for the entire lifetime of the context.

Definition in file source_manager.h.

Macro Definition Documentation

SM_TRACEPOINT

#define SM_TRACEPOINT

(

...

)

Value:

SRN_TRACEPOINT_WITH_GROUP(source_manager __VA_OPT__(, ) __VA_ARGS__)

Definition at line 63 of file source_manager.h.

SRN_SM_NO_ID

#define SRN_SM_NO_ID   UINTPTR_MAX

Definition at line 61 of file source_manager.h.

Typedef Documentation

srn_sm_buffer_id_t

typedef size_t srn_sm_buffer_id_t

Definition at line 66 of file source_manager.h.

srn_sm_buffer_t

typedef struct srn_sm_buffer_t srn_sm_buffer_t

srn_sm_maybe_buffer_t

typedef struct srn_sm_maybe_buffer_t srn_sm_maybe_buffer_t

srn_sm_maybe_id_t

typedef struct srn_sm_maybe_id_t srn_sm_maybe_id_t

srn_sm_t

typedef struct srn_sm_t srn_sm_t

Function Documentation

srn_sm_buffer_data()

const unsigned char * srn_sm_buffer_data ( srn_sm_buffer_t * buf )

inlinestatic

Definition at line 131 of file source_manager.h.

{ return buf->data; }

Here is the caller graph for this function:

srn_sm_buffer_file()

srn_sm_maybe_buffer_t srn_sm_buffer_file	(	srn_sm_t *	sm,
		const char *	filepath,
		bool	reread )

Read and buffer the given filepath into the source manager and return a srn_sm_maybe_buffer_t which either fails or has a pointer to the buffer containing the content of the file.

A nullptr for the buffer means an empty file.

Definition at line 282 of file source_manager.c.

                                                                                          {
  srn_sm_maybe_buffer_t result = {nullptr, nullptr};
  auto maybe_id                = srn_sm_read_buffer_file(sm, filepath, reread);
 
  if (HAS_ERROR(maybe_id)) {
    PUSH_AND_MOVE(sm->ctx, maybe_id, result);
    return result;
  }
 
  // NOTE: buffer can be nullptr, it means an empty file
  result.buffer = srn_sm_get_buffer(sm, maybe_id.id);
 
  return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

srn_sm_buffer_len()

size_t srn_sm_buffer_len ( srn_sm_buffer_t * buf )

inlinestatic

Definition at line 132 of file source_manager.h.

{ return buf->len; }

Here is the caller graph for this function:

srn_sm_get_buffer()

srn_sm_buffer_t * srn_sm_get_buffer	(	srn_sm_t *	sm,
		srn_sm_buffer_id_t	id )

Return the buffer pointer of the given id, nullptr if the id does not exists.

Definition at line 77 of file source_manager.c.

                                                                        {
  if ((id >= 0) && (id < sm->buffer_counts)) {
    SM_TRACEPOINT(get_buffer, id);
    auto maybe_buffer = array_list_get(&sm->buffers, id);
    PANIC_IF(HAS_ERROR(maybe_buffer), "Can't look up the source buffer id");
 
    srn_sm_buffer_t *buffer = maybe_buffer.data;
    return buffer;
  }
 
  PANIC("Source buffer id is not in the correct range");
}

Here is the call graph for this function:

Here is the caller graph for this function:

srn_sm_init()

srn_sm_t * srn_sm_init

(

srn_context_t *

ctx

)

Initialize the source manager, as an immortal data structure.

The buffers themselves will NOT be managed by the the memory manager.

Definition at line 37 of file source_manager.c.

                                          {
 
  srn_sm_t *sm       = srn_mm_immortal_allocate(ctx->engine->mm, srn_sm_t);
  sm->ctx            = ctx;
  sm->buffers        = array_list_empty(ctx);
  sm->buffer_counts  = 0;
  sm->file_to_id_map = hmap_empty(ctx);
 
  srn_fiber_mutex_init(&sm->write_lock);
 
  return sm;
}

Here is the call graph for this function:

Here is the caller graph for this function:

srn_sm_read_buffer_file()

srn_sm_maybe_id_t srn_sm_read_buffer_file	(	srn_sm_t *	sm,
		const char *	filepath,
		bool	reread )

Read and buffer the given filepath into the source manager and return the buffer id.

If the file already exists, it will just return the buffer id, unless the reread is true, which will reread the file and populate the buffer. It will NOT overwrite the old buffer.

Definition at line 174 of file source_manager.c.

                                                                                           {
  srn_sm_maybe_id_t result;
  result.maybe_error = nullptr;
 
  SM_TRACEPOINT(read_buffer_file, filepath, reread);
 
  hmap_key_t k;
  k.data = (void *)filepath;
  k.len  = strnlen(filepath, PATH_MAX);
 
  // Snapshot the map struct under the lock, an insert replaces the two word
  // struct in place and an unsynchronised reader could see a torn root and
  // length pair. CHAMT nodes are immutable, so the walk runs outside it.
  hmap_t snapshot;
  WITH_LOCK(&sm->write_lock, { snapshot = sm->file_to_id_map; });
  auto maybe_buf_id = hmap_lookup(&snapshot, &k, (void *)SRN_SM_NO_ID);
 
  if ((uintptr_t)maybe_buf_id == SRN_SM_NO_ID) {
    // It's a new file
    SM_TRACEPOINT(read_buffer_file_new);
    auto maybe_buffer = read_file(sm, filepath);
 
    if (HAS_ERROR(maybe_buffer)) {
      PUSH_AND_MOVE(sm->ctx, maybe_buffer, result);
      return result;
    }
 
    srn_sm_buffer_t *buf = MALLOC(sm->ctx, srn_sm_buffer_t);
    PANIC_IF_NULL(buf);
 
    buf->data = maybe_buffer.buffer;
    buf->len  = maybe_buffer.len;
    buf->next = nullptr;
    // Buffers live until shutdown, the caller's path string does not.
    buf->filename = srn_dup_str(sm->ctx->engine->mm, filepath);
    PANIC_IF_NULL(buf->filename);
 
    bool lost_race = false;
    WITH_LOCK(&sm->write_lock, {
      // The file was read outside the lock, so another fiber may have loaded
      // the same path in the meantime. Its slot wins and ours is dropped
      // below, which keeps one slot per path.
      void *again = hmap_lookup(&sm->file_to_id_map, &k, (void *)SRN_SM_NO_ID);
      if ((uintptr_t)again != SRN_SM_NO_ID) {
        result.id = (srn_sm_buffer_id_t)(uintptr_t)again;
        lost_race = true;
      } else if (sm->buffer_counts >= UINT16_MAX) {
        // We need to limit the buffer ids to UINT16_MAX since the location stores the id as
        // a 16bit number. If we ever need more than that, we should adjust the limit both here
        // and in the location struct.
        result.maybe_error =
          ERR(sm->ctx, SM_BUFFER_ID_EXHAUSTED, "source manager: buffer ID space exhausted");
      } else {
        // The slot goes in before the map entry, so a failed push leaves no
        // id in the map that points at nothing. No `return` inside this
        // block, the lock is only released after it.
        result.maybe_error = array_list_push(&sm->buffers, buf);
        if (result.maybe_error == nullptr) {
          sm->file_to_id_map = hmap_insert(&sm->file_to_id_map, &k, (void *)sm->buffer_counts);
          result.id          = sm->buffer_counts;
          sm->buffer_counts++;
        }
      }
    });
 
    if (HAS_ERROR(result) || lost_race) {
      srn_mm_free(sm->ctx->engine->mm, buf->data);
      srn_mm_free(sm->ctx->engine->mm, (void *)buf->filename);
      srn_mm_free(sm->ctx->engine->mm, buf);
    }
    return result;
  }
 
  auto buffer_id = (srn_sm_buffer_id_t)maybe_buf_id;
  // We have already read this file once.
  auto buffer = srn_sm_get_buffer(sm, buffer_id);
 
  // At this point buffer being null is a bug
  PANIC_IF_NULL(buffer);
 
  if (reread) {
    auto maybe_new_buffer = read_file(sm, filepath);
    if (HAS_ERROR(maybe_new_buffer)) {
      PUSH_AND_MOVE(sm->ctx, maybe_new_buffer, result);
      return result;
    }
 
    srn_sm_buffer_t *buf = MALLOC(sm->ctx, srn_sm_buffer_t);
    PANIC_IF_NULL(buf);
 
    buf->data     = maybe_new_buffer.buffer;
    buf->len      = maybe_new_buffer.len;
    buf->next     = buffer;
    buf->filename = srn_dup_str(sm->ctx->engine->mm, filepath);
    PANIC_IF_NULL(buf->filename);
 
    WITH_LOCK(&sm->write_lock, {
      /* sm->file_to_id_map = hmap_insert(&sm->file_to_id_map, &k, (void *)buffer_id); */
 
      auto update_err = array_list_update(&sm->buffers, buffer_id, buf);
      PANIC_IF(update_err != nullptr, "failed to update source buffer slot");
    });
  }
 
  result.id = buffer_id;
  return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

srn_sm_shutdown()

void srn_sm_shutdown

(

srn_sm_t *

sm

)

Shutdown the source manager freeing all the buffers.

After this call all the pointers into the buffers will be invalid.

Definition at line 68 of file source_manager.c.

                                   {
  auto err = array_list_for_each(&sm->buffers, free_each_buffer_chain);
  if (err != nullptr) {
    PANIC(err->msg);
  }
  // buffers and file_to_id_map are arena-allocated and freed with the context
  sm->buffer_counts = 0;
}

Here is the call graph for this function:

Here is the caller graph for this function: