Anjuta Developers Reference Manual: Anjuta token stream

## Anjuta token stream Anjuta token stream — Anjuta token stream |

## Stability Level

Unstable, unless otherwise indicated

## Functions

AnjutaTokenStream * | anjuta_token_stream_push ()
AnjutaTokenStream * | anjuta_token_stream_pop ()
AnjutaTokenStream * | anjuta_token_stream_get_parent ()
AnjutaToken * | anjuta_token_stream_get_root ()
GFile * | anjuta_token_stream_get_current_directory ()
GFile * | anjuta_token_stream_get_current_file ()
AnjutaToken * | anjuta_token_stream_tokenize ()
gint | anjuta_token_stream_read ()
void | anjuta_token_stream_append_token ()

## Types and Values

| AnjutaTokenStream

## Includes

    #include <libanjuta/anjuta-token-stream.h>

## Description

A AnjutaTokenStream object reads and writes a list of tokens. It uses two list. The first list is assigned when the object is created. Each token is read as characters discarding the separation between tokens. The second list is written using the data of the first list, so no new memory is allocated, in order to recreate a new list of tokens.

This is used when the lexer needs several passes. At the beginning the file is read as a single token containing the whole file content. The first pass split this content into tokens. Additional passes are done on some parts of the token list to get a more precise splitting.

It is important to not allocate new memory and keep the same character pointers in the additional passes because the token list does not own the memory. The address of each character is used to find the position of the changed data in the file.

Several objects can be linked together to create a stack. It is used for included file or variable expansion.

## Functions

### anjuta_token_stream_push ()

    AnjutaTokenStream *
    anjuta_token_stream_push (AnjutaTokenStream *parent,
                              AnjutaToken *root,
                              AnjutaToken *content,
                              GFile *file);

Create a new stream from a list of tokens. If a parent stream is passed, the new stream keep a link on it, so we can return it when the new stream will be destroyed.

#### Parameters

parent | a parent AnjutaTokenStream object or `NULL`. | [allow-none]
root | a token or `NULL`. | [allow-none]
content | a token list. |
file | a GFile of the file. | [allow-none]

#### Returns

The newly created stream.

### anjuta_token_stream_pop ()

    AnjutaTokenStream *
    anjuta_token_stream_pop (AnjutaTokenStream *stream);

Destroy the stream object and return the parent stream if it exists.

#### Parameters

stream | a AnjutaTokenStream object. |

#### Returns

The parent stream or `NULL` if there is no parent.

[transfer none][allow-none]

### anjuta_token_stream_get_parent ()

    AnjutaTokenStream *
    anjuta_token_stream_get_parent (AnjutaTokenStream *stream);

Return the parent stream

#### Parameters

stream | a AnjutaTokenStream object. |

#### Returns

The parent stream or `NULL` if there is no parent.

[transfer none][allow-none]

### anjuta_token_stream_get_root ()

    AnjutaToken *
    anjuta_token_stream_get_root (AnjutaTokenStream *stream);

Return the root token for the output stream.

#### Parameters

stream | a AnjutaTokenStream object. |

#### Returns

The output root token.

[transfer none]

### anjuta_token_stream_get_current_directory ()

    GFile *
    anjuta_token_stream_get_current_directory
                                   (AnjutaTokenStream *stream);

Return the current directory.

#### Parameters

stream | a AnjutaTokenStream object. |

#### Returns

The current directory.

[transfer none]

### anjuta_token_stream_get_current_file ()

    GFile *
    anjuta_token_stream_get_current_file (AnjutaTokenStream *stream);

Return the current file.

#### Parameters

stream | a AnjutaTokenStream object. |

#### Returns

The current file.

[transfer none]

### anjuta_token_stream_tokenize ()

    AnjutaToken *
    anjuta_token_stream_tokenize (AnjutaTokenStream *stream,
                                  gint type,
                                  gsize length);

Create a token of type from the last length characters previously read and append it in the output stream. The characters are not copied in the output stream, the new token uses the same characters.

#### Parameters

stream | a AnjutaTokenStream object. |
type | a token type. |
length | the token length in character. |

#### Returns

The created token.

### anjuta_token_stream_read ()

    gint
    anjuta_token_stream_read (AnjutaTokenStream *stream,
                              gchar *buffer,
                              gsize max_size);

Read token from the input stream and write the content as a C string in the buffer passed as argument.

#### Parameters

stream | a AnjutaTokenStream object. |
buffer | a character buffer to fill with token data. |
max_size | the size of the buffer. |

#### Returns

The number of characters written in the buffer.

### anjuta_token_stream_append_token ()

    void
    anjuta_token_stream_append_token (AnjutaTokenStream *stream,
                                      AnjutaToken *token);

Append an already existing token in the output stream.

#### Parameters

stream | a AnjutaTokenStream object. |
token | a AnjutaToken object. |

## Types and Values

### AnjutaTokenStream

    typedef struct _AnjutaTokenStream AnjutaTokenStream;