From 3f51f48ee55690c7f64e591a729d9cb034264ca4 Mon Sep 17 00:00:00 2001 From: SAM <8dmasters@gmail.com> Date: Fri, 31 May 2024 10:00:10 +0530 Subject: [PATCH 01/18] Create Transformers.md Added a empty .md file to be served as a guide for Transformers --- contrib/machine-learning/Transformers.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 contrib/machine-learning/Transformers.md diff --git a/contrib/machine-learning/Transformers.md b/contrib/machine-learning/Transformers.md new file mode 100644 index 0000000..e69de29 From b832f8f0c90673462ecc990f45595761565b027d Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Fri, 31 May 2024 13:40:20 +0530 Subject: [PATCH 02/18] Update Transformers.md Added topics --- contrib/machine-learning/Transformers.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/contrib/machine-learning/Transformers.md b/contrib/machine-learning/Transformers.md index e69de29..7bcc102 100644 --- a/contrib/machine-learning/Transformers.md +++ b/contrib/machine-learning/Transformers.md @@ -0,0 +1,21 @@ +# Transformers +## Introduction +A transformer is a deep learning architecture developed by Google and based on the multi-head attention mechanism. It is based on the softmax-based attention +mechanism. Before transformers, predecessors of attention mechanism were added to gated recurrent neural networks, such as LSTMs and gated recurrent units (GRUs), which +processed datasets sequentially. Dependency on previous token computations prevented them from being able to parallelize the attention mechanism. + +## Key Concepts + +## Architecture + +## Implementation +### Theory +Text is converted to numerical representations called tokens, and each token is converted into a vector via looking up from a word embedding table. +At each layer, each token is then contextualized within the scope of the context window with other tokens via a parallel multi-head attention mechanism +allowing the signal for key tokens to be amplified and less important tokens to be diminished. + +### HuggingFace + +### Tensorflow and Keras + +### PyTorch From 5b3c8d7ad108d4a65b4a946d719d242fe9eae308 Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Fri, 31 May 2024 15:14:09 +0530 Subject: [PATCH 03/18] Update Transformers.md Added information about Model Architecture --- contrib/machine-learning/Transformers.md | 59 +++++++++++++++++++++++- 1 file changed, 57 insertions(+), 2 deletions(-) diff --git a/contrib/machine-learning/Transformers.md b/contrib/machine-learning/Transformers.md index 7bcc102..49a1b97 100644 --- a/contrib/machine-learning/Transformers.md +++ b/contrib/machine-learning/Transformers.md @@ -4,9 +4,53 @@ A transformer is a deep learning architecture developed by Google and based on t mechanism. Before transformers, predecessors of attention mechanism were added to gated recurrent neural networks, such as LSTMs and gated recurrent units (GRUs), which processed datasets sequentially. Dependency on previous token computations prevented them from being able to parallelize the attention mechanism. -## Key Concepts +## Model Architecture +

Model Architecture

-## Architecture +### Encoder +The encoder is composed of a stack of identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, positionwise fully connected feed-forward network. Each encoder consists of two major components: a self-attention mechanism and a feed-forward neural network. The self-attention mechanism accepts input encodings from the previous encoder and weights their relevance to each other to generate output encodings. The feed-forward neural network further processes each output encoding individually. These output encodings are then passed to the next encoder as its input, as well as to the decoders. + +### Decoder +The decoder is also composed of a stack of identical layers. In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack. The decoder functions in a similar fashion to the encoder, but an additional attention mechanism is inserted which instead draws relevant information from the encodings generated by the encoders. This mechanism can also be called the encoder-decoder attention. + +### Attention +#### Scaled Dot-Product Attention +The input consists of queries and keys of dimension dk, and values of dimension dv. We compute the dot products of the query with all keys, divide each by √dk, and apply a softmax function to obtain the weights on the values. + +> Attention(Q, K, V) = softmax(QKT / √dk) * V + +#### Multi-Head Attention +Instead of performing a single attention function with dmodel-dimensional keys, values and queries, it is beneficial to linearly project the queries, keys and values h times with different, learned linear projections to dk, dk and dv dimensions, respectively. + +Multi-head attention allows the model to jointly attend to information from different representation +subspaces at different positions. With a single attention head, averaging inhibits this. + +> MultiHead(Q, K, V) = Concat(head1, ..., headh) * WO + +where, + +> headi = Attention(Q * WiQ, K * WiK, V * WiV) + +where the projections are parameter matrices. + +#### Masked Attention +It may be necessary to cut out attention links between some word-pairs. For example, the decoder for token position +𝑡 should not have access to token position 𝑡+1. + +> MaskedAttention(Q, K, V) = softmax(M + (QKT / √dk)) * V + +### Feed-Forward Network +Each of the layers in the encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This +consists of two linear transformations with a ReLU activation in between. +> FFN(x) = (max(0, (x * W1) + b1) * W2) + b2 + +### Positional Encoding +A positional encoding is a fixed-size vector representation that encapsulates the relative positions of tokens within a target sequence: it provides the transformer model with information about where the words are in the input sequence. + +The sine and cosine functions of different frequencies: +> PE(pos,2i) = sin(pos/100002i/dmodel) + +> PE(pos,2i+1) = cos(pos/100002i/dmodel) ## Implementation ### Theory @@ -19,3 +63,14 @@ allowing the signal for key tokens to be amplified and less important tokens to ### Tensorflow and Keras ### PyTorch + +## Application +The transformer has had great success in natural language processing (NLP). Many large language models such as GPT-2, GPT-3, GPT-4, Claude, BERT, XLNet, RoBERTa and ChatGPT demonstrate the ability of transformers to perform a wide variety of such NLP-related tasks, and have the potential to find real-world applications. + +These may include: +- Machine translation +- Document summarization +- Text generation +- Biological sequence analysis +- Computer code generation +- Video analysis From 35357d2fe3f6650faca3d0f19327fe4fa7e8629d Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Fri, 31 May 2024 17:04:07 +0530 Subject: [PATCH 04/18] Update Transformers.md Added Tensorflow implementation --- contrib/machine-learning/Transformers.md | 136 ++++++++++++++++++++++- 1 file changed, 134 insertions(+), 2 deletions(-) diff --git a/contrib/machine-learning/Transformers.md b/contrib/machine-learning/Transformers.md index 49a1b97..a5a56ec 100644 --- a/contrib/machine-learning/Transformers.md +++ b/contrib/machine-learning/Transformers.md @@ -58,12 +58,144 @@ Text is converted to numerical representations called tokens, and each token is At each layer, each token is then contextualized within the scope of the context window with other tokens via a parallel multi-head attention mechanism allowing the signal for key tokens to be amplified and less important tokens to be diminished. -### HuggingFace +### Tensorflow +Tensorflow provides the transformer encoder and decoder block that can be implemented by the specification of the user. Although, the transformer is not provided as a standalone to be imported and executed, the user has to create the model first. They also have a tutorial on how to implement the transformer from scratch for machine translation and can be found [here](https://www.tensorflow.org/text/tutorials/transformer). -### Tensorflow and Keras +More information on [encoder](https://www.tensorflow.org/api_docs/python/tfm/nlp/layers/TransformerEncoderBlock) and [decoder](https://www.tensorflow.org/api_docs/python/tfm/nlp/layers/TransformerDecoderBlock) block mentioned in the code. + +Imports: +``` +import tensorflow as tf +import tensorflow_models as tfm +``` + +Adding word embeddings and positional encoding: +``` +class PositionalEmbedding(tf.keras.layers.Layer): + def __init__(self, vocab_size, d_model): + super().__init__() + self.d_model = d_model + self.embedding = tf.keras.layers.Embedding(vocab_size, d_model, mask_zero=True) + self.pos_encoding = tfm.nlp.layers.RelativePositionEmbedding(hidden_size=d_model) + + def compute_mask(self, *args, **kwargs): + return self.embedding.compute_mask(*args, **kwargs) + + def call(self, x): + length = tf.shape(x)[1] + x = self.embedding(x) + x = x + self.pos_encoding[tf.newaxis, :length, :] + return x +``` + +Creating the encoder for the transformer: +``` +class Encoder(tf.keras.layers.Layer): + def __init__(self, num_layers, d_model, num_heads, + dff, vocab_size, dropout_rate=0.1): + super().__init__() + + self.d_model = d_model + self.num_layers = num_layers + + self.pos_embedding = PositionalEmbedding( + vocab_size=vocab_size, d_model=d_model) + + self.enc_layers = [ + tfm.nlp.layers.TransformerEncoderBlock(output_last_dim=d_model, + num_attention_heads=num_heads, + inner_dim=dff, + inner_activation="relu", + inner_dropout=dropout_rate) + for _ in range(num_layers)] + self.dropout = tf.keras.layers.Dropout(dropout_rate) + + def call(self, x): + x = self.pos_embedding(x, length=2048) + x = self.dropout(x) + + for i in range(self.num_layers): + x = self.enc_layers[i](x) + + return x +``` + +Creating the decoder for the transformer: +``` +class Decoder(tf.keras.layers.Layer): + def __init__(self, num_layers, d_model, num_heads, dff, vocab_size, + dropout_rate=0.1): + super(Decoder, self).__init__() + + self.d_model = d_model + self.num_layers = num_layers + + self.pos_embedding = PositionalEmbedding(vocab_size=vocab_size, + d_model=d_model) + self.dropout = tf.keras.layers.Dropout(dropout_rate) + self.dec_layers = [ + tfm.nlp.layers.TransformerDecoderBlock(num_attention_heads=num_heads, + intermediate_size=dff, + intermediate_activation="relu", + dropout_rate=dropout_rate) + for _ in range(num_layers)] + + def call(self, x, context): + x = self.pos_embedding(x) + x = self.dropout(x) + + for i in range(self.num_layers): + x = self.dec_layers[i](x, context) + + return x +``` + +Combining the encoder and decoder to create the transformer: +``` +class Transformer(tf.keras.Model): + def __init__(self, num_layers, d_model, num_heads, dff, + input_vocab_size, target_vocab_size, dropout_rate=0.1): + super().__init__() + self.encoder = Encoder(num_layers=num_layers, d_model=d_model, + num_heads=num_heads, dff=dff, + vocab_size=input_vocab_size, + dropout_rate=dropout_rate) + + self.decoder = Decoder(num_layers=num_layers, d_model=d_model, + num_heads=num_heads, dff=dff, + vocab_size=target_vocab_size, + dropout_rate=dropout_rate) + + self.final_layer = tf.keras.layers.Dense(target_vocab_size) + + def call(self, inputs): + context, x = inputs + + context = self.encoder(context) + x = self.decoder(x, context) + logits = self.final_layer(x) + + return logits +``` + +Model initialization that be used for training and inference: +``` +transformer = Transformer( + num_layers=num_layers, + d_model=d_model, + num_heads=num_heads, + dff=dff, + input_vocab_size=tokenizers.pt.get_vocab_size().numpy(), + target_vocab_size=tokenizers.en.get_vocab_size().numpy(), + dropout_rate=dropout_rate +) +``` ### PyTorch + +### HuggingFace + ## Application The transformer has had great success in natural language processing (NLP). Many large language models such as GPT-2, GPT-3, GPT-4, Claude, BERT, XLNet, RoBERTa and ChatGPT demonstrate the ability of transformers to perform a wide variety of such NLP-related tasks, and have the potential to find real-world applications. From f56131349d1ad0e6e27209ea3d9217c13e695e8e Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Fri, 31 May 2024 17:13:19 +0530 Subject: [PATCH 05/18] Update Transformers.md Added python tag to code blocks --- contrib/machine-learning/Transformers.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/contrib/machine-learning/Transformers.md b/contrib/machine-learning/Transformers.md index a5a56ec..ffdea05 100644 --- a/contrib/machine-learning/Transformers.md +++ b/contrib/machine-learning/Transformers.md @@ -64,13 +64,13 @@ Tensorflow provides the transformer encoder and decoder block that can be implem More information on [encoder](https://www.tensorflow.org/api_docs/python/tfm/nlp/layers/TransformerEncoderBlock) and [decoder](https://www.tensorflow.org/api_docs/python/tfm/nlp/layers/TransformerDecoderBlock) block mentioned in the code. Imports: -``` +```python import tensorflow as tf import tensorflow_models as tfm ``` Adding word embeddings and positional encoding: -``` +```python class PositionalEmbedding(tf.keras.layers.Layer): def __init__(self, vocab_size, d_model): super().__init__() @@ -89,7 +89,7 @@ class PositionalEmbedding(tf.keras.layers.Layer): ``` Creating the encoder for the transformer: -``` +```python class Encoder(tf.keras.layers.Layer): def __init__(self, num_layers, d_model, num_heads, dff, vocab_size, dropout_rate=0.1): @@ -121,7 +121,7 @@ class Encoder(tf.keras.layers.Layer): ``` Creating the decoder for the transformer: -``` +```python class Decoder(tf.keras.layers.Layer): def __init__(self, num_layers, d_model, num_heads, dff, vocab_size, dropout_rate=0.1): @@ -151,7 +151,7 @@ class Decoder(tf.keras.layers.Layer): ``` Combining the encoder and decoder to create the transformer: -``` +```python class Transformer(tf.keras.Model): def __init__(self, num_layers, d_model, num_heads, dff, input_vocab_size, target_vocab_size, dropout_rate=0.1): @@ -179,7 +179,7 @@ class Transformer(tf.keras.Model): ``` Model initialization that be used for training and inference: -``` +```python transformer = Transformer( num_layers=num_layers, d_model=d_model, From ca9c6a9a1f23d1e6b5901954c8a5c5c772bd80a7 Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Fri, 31 May 2024 17:44:28 +0530 Subject: [PATCH 06/18] Update Transformers.md Added PyTorch implementation --- contrib/machine-learning/Transformers.md | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/contrib/machine-learning/Transformers.md b/contrib/machine-learning/Transformers.md index ffdea05..cb90cf1 100644 --- a/contrib/machine-learning/Transformers.md +++ b/contrib/machine-learning/Transformers.md @@ -56,7 +56,9 @@ The sine and cosine functions of different frequencies: ### Theory Text is converted to numerical representations called tokens, and each token is converted into a vector via looking up from a word embedding table. At each layer, each token is then contextualized within the scope of the context window with other tokens via a parallel multi-head attention mechanism -allowing the signal for key tokens to be amplified and less important tokens to be diminished. +allowing the signal for key tokens to be amplified and less important tokens to be diminished. + +The transformer uses an encoder-decoder architecture. The encoder extracts features from an input sentence, and the decoder uses the features to produce an output sentence. Some architectures use full encoders and decoders, autoregressive encoders and decoders, or combination of both. This depends on the usage and context of the input. ### Tensorflow Tensorflow provides the transformer encoder and decoder block that can be implemented by the specification of the user. Although, the transformer is not provided as a standalone to be imported and executed, the user has to create the model first. They also have a tutorial on how to implement the transformer from scratch for machine translation and can be found [here](https://www.tensorflow.org/text/tutorials/transformer). @@ -192,10 +194,30 @@ transformer = Transformer( ``` ### PyTorch +Unlike Tensorflow, PyTorch provides the full implementation of the transformer model that can be executed on the go. More information can be found [here](https://pytorch.org/docs/stable/_modules/torch/nn/modules/transformer.html#Transformer). A full implementation of the model can be found [here](https://github.com/pytorch/examples/tree/master/word_language_model). +Imports: +```python +import torch +import torch.nn as nn +``` + +Initializing the model: +```python +transformer = nn.Transformer(nhead=16, num_encoder_layers=8) +``` + +Sample Implementation: +```python +src = torch.rand((10, 32, 512)) +tgt = torch.rand((20, 32, 512)) + +output = transformer(src, tgt) +``` ### HuggingFace + ## Application The transformer has had great success in natural language processing (NLP). Many large language models such as GPT-2, GPT-3, GPT-4, Claude, BERT, XLNet, RoBERTa and ChatGPT demonstrate the ability of transformers to perform a wide variety of such NLP-related tasks, and have the potential to find real-world applications. From d81e875a3235d7c9f50f9a42bb359bf80d3146a1 Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Fri, 31 May 2024 18:32:47 +0530 Subject: [PATCH 07/18] Update Transformers.md Added outputs for PyTorch and HuggingFace implementation --- contrib/machine-learning/Transformers.md | 114 ++++++++++++++++++++++- 1 file changed, 112 insertions(+), 2 deletions(-) diff --git a/contrib/machine-learning/Transformers.md b/contrib/machine-learning/Transformers.md index cb90cf1..e9e50e8 100644 --- a/contrib/machine-learning/Transformers.md +++ b/contrib/machine-learning/Transformers.md @@ -61,6 +61,8 @@ allowing the signal for key tokens to be amplified and less important tokens to The transformer uses an encoder-decoder architecture. The encoder extracts features from an input sentence, and the decoder uses the features to produce an output sentence. Some architectures use full encoders and decoders, autoregressive encoders and decoders, or combination of both. This depends on the usage and context of the input. ### Tensorflow +TensorFlow is a free and open-source software library for machine learning and artificial intelligence. It can be used across a range of tasks but has a particular focus on training and inference of deep neural networks. It was developed by the Google Brain team for Google's internal use in research and production. + Tensorflow provides the transformer encoder and decoder block that can be implemented by the specification of the user. Although, the transformer is not provided as a standalone to be imported and executed, the user has to create the model first. They also have a tutorial on how to implement the transformer from scratch for machine translation and can be found [here](https://www.tensorflow.org/text/tutorials/transformer). More information on [encoder](https://www.tensorflow.org/api_docs/python/tfm/nlp/layers/TransformerEncoderBlock) and [decoder](https://www.tensorflow.org/api_docs/python/tfm/nlp/layers/TransformerDecoderBlock) block mentioned in the code. @@ -193,7 +195,22 @@ transformer = Transformer( ) ``` +Sample: +```python +src = tf.random.uniform((64, 40)) +tgt = tf.random.uniform((64, 50)) + +output = transformer((src, tgt)) +``` + +O/P: +``` + +``` + ### PyTorch +PyTorch is a machine learning library based on the Torch library, used for applications such as computer vision and natural language processing, originally developed by Meta AI and now part of the Linux Foundation umbrella. + Unlike Tensorflow, PyTorch provides the full implementation of the transformer model that can be executed on the go. More information can be found [here](https://pytorch.org/docs/stable/_modules/torch/nn/modules/transformer.html#Transformer). A full implementation of the model can be found [here](https://github.com/pytorch/examples/tree/master/word_language_model). Imports: @@ -207,7 +224,7 @@ Initializing the model: transformer = nn.Transformer(nhead=16, num_encoder_layers=8) ``` -Sample Implementation: +Sample: ```python src = torch.rand((10, 32, 512)) tgt = torch.rand((20, 32, 512)) @@ -215,8 +232,101 @@ tgt = torch.rand((20, 32, 512)) output = transformer(src, tgt) ``` -### HuggingFace +O/P: +``` +tensor([[[ 0.2938, -0.4824, -0.7816, ..., 0.0742, 0.5162, 0.3632], + [-0.0786, -0.5241, 0.6384, ..., 0.3462, -0.0618, 0.9943], + [ 0.7827, 0.1067, -0.1637, ..., -1.7730, -0.3322, -0.0029], + ..., + [-0.3202, 0.2341, -0.0896, ..., -0.9714, -0.1251, -0.0711], + [-0.1663, -0.5047, -0.0404, ..., -0.9339, 0.3963, 0.1018], + [ 1.2834, -0.4400, 0.0486, ..., -0.6876, -0.4752, 0.0180]], + [[ 0.9869, -0.7384, -1.0704, ..., -0.9417, 1.3279, -0.1665], + [ 0.3445, -0.2454, -0.3644, ..., -0.4856, -1.1004, -0.6819], + [ 0.7568, -0.3151, -0.5034, ..., -1.2081, -0.7119, 0.3775], + ..., + [-0.0451, -0.7596, 0.0168, ..., -0.8267, -0.3272, 1.0457], + [ 0.3150, -0.6588, -0.1840, ..., 0.1822, -0.0653, 0.9053], + [ 0.8692, -0.3519, 0.3128, ..., -1.8446, -0.2325, -0.8662]], + + [[ 0.9719, -0.3113, 0.4637, ..., -0.4422, 1.2348, 0.8274], + [ 0.3876, -0.9529, -0.7810, ..., -0.5843, -1.1439, -0.3366], + [-0.5774, 0.3789, -0.2819, ..., -1.4057, 0.4352, 0.1474], + ..., + [ 0.6899, -0.1146, -0.3297, ..., -1.7059, -0.1750, 0.4203], + [ 0.3689, -0.5174, -0.1253, ..., 0.1417, 0.4159, 0.7560], + [ 0.5024, -0.7996, 0.1592, ..., -0.8344, -1.1125, 0.4736]], + + ..., + + [[ 0.0704, -0.3971, -0.2768, ..., -1.9929, 0.8608, 1.2264], + [ 0.4013, -0.0962, -0.0965, ..., -0.4452, -0.8682, -0.4593], + [ 0.1656, 0.5224, -0.1723, ..., -1.5785, 0.3219, 1.1507], + ..., + [-0.9443, 0.4653, 0.2936, ..., -0.9840, -0.0142, -0.1595], + [-0.6544, -0.3294, -0.0803, ..., 0.1623, -0.5061, 0.9824], + [-0.0978, -1.0023, -0.6915, ..., -0.2296, -0.0594, -0.4715]], + + [[ 0.6531, -0.9285, -0.0331, ..., -1.1481, 0.7768, -0.7321], + [ 0.3325, -0.6683, -0.6083, ..., -0.4501, 0.2289, 0.3573], + [-0.6750, 0.4600, -0.8512, ..., -2.0097, -0.5159, 0.2773], + ..., + [-1.4356, -1.0135, 0.0081, ..., -1.2985, -0.3715, -0.2678], + [ 0.0546, -0.2111, -0.0965, ..., -0.3822, -0.4612, 1.6217], + [ 0.7700, -0.5309, -0.1754, ..., -2.2807, -0.0320, -1.5551]], + + [[ 0.2399, -0.9659, 0.1086, ..., -1.1756, 0.4063, 0.0615], + [-0.2202, -0.7972, -0.5024, ..., -0.9126, -1.5248, 0.2418], + [ 0.5215, 0.4540, 0.0036, ..., -0.2135, 0.2145, 0.6638], + ..., + [-0.2190, -0.4967, 0.7149, ..., -0.3324, 0.3502, 1.0624], + [-0.0108, -0.9205, -0.1315, ..., -1.0153, 0.2989, 1.1415], + [ 1.1284, -0.6560, 0.6755, ..., -1.2157, 0.8580, -0.5022]]], + grad_fn=) +``` +``` +>> output.shape +torch.Size([20, 32, 512]) +``` + +### HuggingFace +Hugging Face, Inc. is a French-American company incorporated under the Delaware General Corporation Law and based in New York City that develops computation tools for building applications using machine learning. + +It has a wide-range of models that can implemented in Tensorflow, PyTorch and other development backends as well. The models are already trained on a dataset and can be pretrained on custom dataset for customized use, according to the user. The information for training the model and loading the pretrained model can be found [here](https://huggingface.co/docs/transformers/en/training). + +In HuggingFace, `pipeline` is used to run inference from the trained model available in the Hub. This is very beginner friendly. The model is downloaded to the local system on running the script before running the inference. It has to be made sure that the model downloaded does not exceed your available data plan. + +Imports: +```python +from transformers import pipeline +``` + +Initialization: + +The model used here is BART (large) which was trained on MultiNLI dataset, which consist of sentence paired with its textual entailment. +```python +classifier = pipeline(model="facebook/bart-large-mnli") +``` + +Sample: + +The first argument is the sentence which needs to be analyzed. The second argument, `candidate_labels`, is the list of labels which most likely the first argument sentence belongs to. The output dictionary will have a key as `score`, where the highest index is the textual entailment of the sentence with the index of the label in the list. + +```python +output = classifier( + "I need to leave but later", + candidate_labels=["urgent", "not urgent", "sleep"], +) +``` + +O/P: + +``` +{'sequence': 'I need to leave but later', + 'labels': ['not urgent', 'urgent', 'sleep'], + 'scores': [0.8889380097389221, 0.10631518065929413, 0.00474683940410614]} +``` ## Application The transformer has had great success in natural language processing (NLP). Many large language models such as GPT-2, GPT-3, GPT-4, Claude, BERT, XLNet, RoBERTa and ChatGPT demonstrate the ability of transformers to perform a wide variety of such NLP-related tasks, and have the potential to find real-world applications. From 251c4d86894b26e113b1b4426aeaf412c0b4c977 Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Fri, 31 May 2024 18:43:19 +0530 Subject: [PATCH 08/18] Update Transformers.md Added Tensorflow output and fixed wrong mentions --- contrib/machine-learning/Transformers.md | 93 +++++++++++++++++++++++- 1 file changed, 91 insertions(+), 2 deletions(-) diff --git a/contrib/machine-learning/Transformers.md b/contrib/machine-learning/Transformers.md index e9e50e8..d30bd63 100644 --- a/contrib/machine-learning/Transformers.md +++ b/contrib/machine-learning/Transformers.md @@ -189,8 +189,8 @@ transformer = Transformer( d_model=d_model, num_heads=num_heads, dff=dff, - input_vocab_size=tokenizers.pt.get_vocab_size().numpy(), - target_vocab_size=tokenizers.en.get_vocab_size().numpy(), + input_vocab_size=64, + target_vocab_size=64, dropout_rate=dropout_rate ) ``` @@ -205,7 +205,96 @@ output = transformer((src, tgt)) O/P: ``` + +``` +``` +>>> output.shape +TensorShape([64, 50, 64]) ``` ### PyTorch From 44303ff8e04e496e0f545d92a60e9758b971e314 Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Fri, 31 May 2024 18:45:07 +0530 Subject: [PATCH 09/18] Update Transformers.md Extended Introduction --- contrib/machine-learning/Transformers.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/contrib/machine-learning/Transformers.md b/contrib/machine-learning/Transformers.md index d30bd63..97346bd 100644 --- a/contrib/machine-learning/Transformers.md +++ b/contrib/machine-learning/Transformers.md @@ -1,8 +1,9 @@ # Transformers ## Introduction A transformer is a deep learning architecture developed by Google and based on the multi-head attention mechanism. It is based on the softmax-based attention -mechanism. Before transformers, predecessors of attention mechanism were added to gated recurrent neural networks, such as LSTMs and gated recurrent units (GRUs), which -processed datasets sequentially. Dependency on previous token computations prevented them from being able to parallelize the attention mechanism. +mechanism. Before transformers, predecessors of attention mechanism were added to gated recurrent neural networks, such as LSTMs and gated recurrent units (GRUs), which processed datasets sequentially. Dependency on previous token computations prevented them from being able to parallelize the attention mechanism. + +Transformers are a revolutionary approach to natural language processing (NLP). Unlike older models, they excel at understanding long-range connections between words. This "attention" mechanism lets them grasp the context of a sentence, making them powerful for tasks like machine translation, text summarization, and question answering. Introduced in 2017, transformers are now the backbone of many large language models, including tools you might use every day. Their ability to handle complex relationships in language is fueling advancements in AI across various fields. ## Model Architecture

Model Architecture

From 402289d3ceae6add26458f41d33364eff322f8e2 Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Sun, 2 Jun 2024 10:23:09 +0530 Subject: [PATCH 10/18] Rename Transformers.md to transformers.md --- contrib/machine-learning/{Transformers.md => transformers.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename contrib/machine-learning/{Transformers.md => transformers.md} (100%) diff --git a/contrib/machine-learning/Transformers.md b/contrib/machine-learning/transformers.md similarity index 100% rename from contrib/machine-learning/Transformers.md rename to contrib/machine-learning/transformers.md From e13fb234f2829a1b918aac3c044e4feef5066d34 Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Sun, 2 Jun 2024 10:23:56 +0530 Subject: [PATCH 11/18] Add transformers.md --- contrib/machine-learning/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/contrib/machine-learning/index.md b/contrib/machine-learning/index.md index b6945cd..13d9440 100644 --- a/contrib/machine-learning/index.md +++ b/contrib/machine-learning/index.md @@ -16,3 +16,4 @@ - [Types_of_Cost_Functions](cost-functions.md) - [Clustering](clustering.md) - [Grid Search](grid-search.md) +- [Transformers](transformers.md) From 229458043245de432c96f05457dd79330d568a92 Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Sun, 2 Jun 2024 10:39:37 +0530 Subject: [PATCH 12/18] Update transformers.md Added source for the image --- contrib/machine-learning/transformers.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/contrib/machine-learning/transformers.md b/contrib/machine-learning/transformers.md index 97346bd..9fbfa6d 100644 --- a/contrib/machine-learning/transformers.md +++ b/contrib/machine-learning/transformers.md @@ -6,7 +6,10 @@ mechanism. Before transformers, predecessors of attention mechanism were added t Transformers are a revolutionary approach to natural language processing (NLP). Unlike older models, they excel at understanding long-range connections between words. This "attention" mechanism lets them grasp the context of a sentence, making them powerful for tasks like machine translation, text summarization, and question answering. Introduced in 2017, transformers are now the backbone of many large language models, including tools you might use every day. Their ability to handle complex relationships in language is fueling advancements in AI across various fields. ## Model Architecture -

Model Architecture

+

+ Model Architecture
+Source: Attention Is All You Need - Figure 1 +

### Encoder The encoder is composed of a stack of identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, positionwise fully connected feed-forward network. Each encoder consists of two major components: a self-attention mechanism and a feed-forward neural network. The self-attention mechanism accepts input encodings from the previous encoder and weights their relevance to each other to generate output encodings. The feed-forward neural network further processes each output encoding individually. These output encodings are then passed to the next encoder as its input, as well as to the decoders. From 07a1e737606d10d7e96e2291cec21003309bfbf0 Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Sun, 2 Jun 2024 10:46:21 +0530 Subject: [PATCH 13/18] Update transformers.md Added bibliography --- contrib/machine-learning/transformers.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/contrib/machine-learning/transformers.md b/contrib/machine-learning/transformers.md index 9fbfa6d..0799b7f 100644 --- a/contrib/machine-learning/transformers.md +++ b/contrib/machine-learning/transformers.md @@ -430,4 +430,11 @@ These may include: - Text generation - Biological sequence analysis - Computer code generation -- Video analysis + +## Bibliography +- [Attention Is All You Need](https://arxiv.org/pdf/1706.03762) +- [Tensorflow Tutorial](https://www.tensorflow.org/text/tutorials/transformer) +- [Tensorflow Models Docs](https://www.tensorflow.org/api_docs/python/tfm/nlp/layers) +- [Wikipedia](https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)) +- [HuggingFace](https://huggingface.co/docs/transformers/en/index) +- [PyTorch](https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html) From 3e35783d59919de603e6d11e81df54fdf47662b2 Mon Sep 17 00:00:00 2001 From: SAM <8dmasters@gmail.com> Date: Sun, 2 Jun 2024 16:19:11 +0530 Subject: [PATCH 14/18] Create transformer-architecture.png --- .../assets/transformer-architecture.png | Bin 0 -> 57018 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 contrib/machine-learning/assets/transformer-architecture.png diff --git a/contrib/machine-learning/assets/transformer-architecture.png b/contrib/machine-learning/assets/transformer-architecture.png new file mode 100644 index 0000000000000000000000000000000000000000..2854ab0a8d3e6f0ba0c53735afb57e81eb825a3f GIT binary patch literal 57018 zcmdqIRZv|)*DV?%xVsbF-95qG-7UDgLy(QT2A5#L-66OJx1hn@-S66bztpMobgS;u zO;rk5d+jB&yXPEpjL~5Va^eWExUlcuy+e?c5K(&f4&vzDyZ3_7kie08^8;_-2ZWQ7 zxX`=s${TdPs6MPR7eE!rEr69}OZ{1Y}tiUgkxW5DbfIqW*2gd=N z(NhdTxA1lXWhc^tL!`_Cq5f z4fKQ{IULT>pEp@dQ$!N-;}*$hC&xUo`uh4ZadM_e!vuDA3aXW95T{^^u^V=oVpB~1 zZLq=PaXkwC{sBrzSy_2zQo%hMpWA+;7bcQOP!HO{8wLjEtAT+*2AwNN5z8GBe7|5@ z=4Pke{*e^M-12hzd9iP=FV99AE$Wqeds77w*gW&Ivq{5O^Hc{_XAj zJkERm)XIhbUY;L~xE^ngm-0S*!E-?#gk{vKD|Yvtuh6YF9*CyeZFAfXM|S7u=eJ&N zLYc4B7cY{_Wb(K$F&R#j6bLs|foiD20ZFP1I&!eFMXHvCJzO7d9{sH~%I>{Y=M4yA zXJbQG{iV|NOSLrU9iv9omr(k8*;FQo=|Wkwp2JLDxBe34qKQ^FyN@3~hVJ^^Uu?(h z{)~Q~nb4P}sH*XHY4g>zpli1YCho9<&Z$N_{h!h)atsg_Q$M?a`#7w3e`*F9&JbZ* zgA7qI5D*Z&pY9YH`XVbU8ClH6=u$T1`ob}`+p`lH^$CHON|1)ne*sV7gyNs_*@UO7 z?Ge~n(P7dl47oN-4UW`2CIzpEj%<%fRaH#Py99WCJd{1fxZt@@Mze*`Vf~k-x^HV! zO8OTjgHGtqpY({D@QVm_prmgy-;JXtG(~};xXYq8_$ooP44G;I`Otz zug_3WP$5+BkB^S%K5D_VLzb6TXg5|+-5&i#o12?6%09cfiFTE{V2LRa?qd4&p7GEK zn?f>28v(&*JeB#wcosj}ds)I23JQwBH1<;OM3TYbpOOh@8+~x-)Jh_w^7EzY@}c(C zMt!{==3#S}yW`9*2P(v?CI!q`;F}t3go{<2b z-D;a7LeEbb@bDU9iy{0Yb^v~Y6hHhQ2Yg3PZBfwCxAhgXIGM*$n)w@^^(8|OjZ%C4 z>4CBK9hokY7rlsf+#VLSCYMeYDN!oez@U$u<5RXz@Cy#Acix}sUZ>*_@OcRO8VDKY zBh@Tf!nN=5Z8#CdsupG#QS7@);zY|CFr`PxL5c+6d*73yJ8yLbLZK`rP|AzEn4|VI z+N}+Xmk^?JeVZv(blj&*q)lh>M6tECb$`06UZ8xq+7G1DtQq|83xu(<>Bvm}rh;sW zii)bzN;>Zx+9>G7lNt;l|6W98ELFF|A~ zSuFm*WWB&tT^W?6U7|!zT)`$H$6d>)-$7>-p}aKugOwE2D>gwu15J|N*P(hMRKN$&tq+8b1)bx}JBv_;&$Ecd-*4Ed-Ox|0h zp@S!2SQ!ikWG1kfL)fWf&Gu37BkhQ*+{GTxsmaEy3A1p7K!D2rFk=z{1AkInSu zNLuJ|bG6-T!y|t7nUs!BW@fG3>ox=!y#>lbxy*;2o*v;N`5!FX1IJ5^eF9&{GdXM) zV}LyV0u2xUMC?Mbh51QId^=>OBvu+ep{gnv7zrX~R@OkWK}C+5$e}zACMM*8*d$p= zUaCusYVZ3C^UDEe>%TSMNO#(SOpI;Q;Xazm%;kQTvp~e}>D(LJ;M|ESd<;YtJ(-ml z#ym**M|=ji6D(!TomPvCi{P1mYW=e8K^&xB;rrOqEmey#>B+TV_<1$XfU7sm9$Wlv5?8* z5>QxJh^&BmzBv#R!4c5=0Ms}~Nrs9LB~Kuv^IB!!?b-%A5E51|$r4_bKxGiz_5 zI^Vp=@W$)*B#eW{bhAHdFpHlTpMapRjal=+YOV~WyMoVjqc^PE=}A38ki%RflLw`a zEpjY3C`*1f7g5;2Ajdqv`8-!RB+uh=CvTeBW|56q>Wj^c$3nXor{~okarSpZR7^zK zVWv?#Vq&k?7k*ewI;wd9Y$OCFB}c@c}Y4*E(RQ zl{!x=43)f>+dMUk&JHL-DggfJw(N8{#C&>wR_yfkS)WO!l+OySU%29H5op0|fpp}_ z|7Q7--wY_K_HJ$=Kw;c7k4;!!bH&7_VE7-{D7^8%I5IzZqK&2JCj~gvioE7y8FQwW zk%S3p;A$y$oKv8R{+J%`$RuO=9Y9vt^~NX5&E3O=XGQP{S_X_j(U^gS#rg$QdQguA zH3mIYzKkw>!jvH#_}qXw5!z#T*hEzB3!I*xCJWub6>V~u3eoo?-zUdvBm`$N>IlYS-EzjoQPchv0K~%HU;@_@FN!^pc^GBInwNwQ75}6 zP{@+RhcU@Wfe{~eJ#mAf9U-Eb(ue=|82P(0HYm*B77ED5UDt>6=j%OC&1%$}ikAQSDSpQ3WEQ7nR#UXsGpXodpd`pNGSg{yF`n(Wk7^rfTq9Y?<$^EJ ziX*yvc^N;Ih3zqyQAW7D|8ja*NRC(SGHa?*qV(zJ=H{t`C{32LR+Lpy92jG8g>ncq^*bpc(yg1VK{A74r&??LNP5CDGN+>-?R|_TjW+p>%s);ab@*P#nrr!}F=GVvS zd5uB^sFZg$IY zW0$%WhS;;7%7OtoD#=v7Wgo zg*+XY$ZTE+MxCr4dDO|9b{-P(lZ?HIBD6FYbIANWG4ttqdZ}bu@~m_LV^3#-WrV&F zbn!LC=kvuCG=9+am&vKV31f44J~v;6XeX~5RAbX<;UU-pg8>sG-2vmr__i)mVW&y5Wj`V6F{#9jXIg35}$xx@Sq2ch`g}zY4GNs-ON=-G8!@`O~*( z?zNQ8_00n{w=36tuA19%=T8`fF_cl*5J^x-$jG=UF!zNMSl~%PIWEr5y^g<(Ips;j zSMeRCyT45SIazConLi;M{8192$k*uv*oDb*yxQZg ze70hNn(Ee2wm4Tqfn++Q$Hg{AvcT6yVXcFt5vE4$4tZ$3<_@7E&*M01Ws1AII}yBo zv((PF>`SZypcjk)7N@_~b;p0@Yz;R)v{ycI9uEvRk0`XcI{;vmY|g_KbdLru2+){} zdho`^#^zR5`Kl=#M!IZhf3gaQQsc4fE=?yPv-o6BBckWS^Xw5lQE;Q7F@3 zCWJNHKmea>3ANI|<90$P_CW*s$IScx9NJE%D4ktdA@I1^inGF@_|}6)0~9xcEqrmO z6I!*hSSxIb?H)|vK=!+{^}vOqZ%V;%SfJWkR^Du8YQnby3mS+af)lR=;JyJ+&onx`xgB@MqK1=b z^8w6vp$MPAvJUP9XF&1f(s|!Tk|(C8(YEoq9doSaE2tE5{Ha*~zbd%@mKyDvyzgtE zUnLC%RWV>QgdkL?hBbe%eB08RCG~l@+M6xKvYaU<&Z}XT@qVkR$b^K^{ZV*5b8~7H zhN$ngNaeb8feJd1_oo*Q{B|D!w3*89)nI~Q`sK7EFpe@Nce(qC78t+YlgxA!RMbe9 z!#Ts=P!vbFp58aW@ud+$O7s5uP`ym69fMxmEm1r5?#pRuK_A0t28SN}T5C&-qYB4P z_WC){CX4LQrq#-&bI{YM{GyqsU?39o{e5Z1;7Cm~HWq^nbciW4Sg5E9YFh2P&g7fVhfrgQ$*uMRvSmfv0EWB`|+C?aF*CYtI=i0tdml9 zU;LJXd3RH_Os zThl$HtV2Eb5r&nVd8jyrTIh3vaM`{OpQC&B0w)WLRI-(U5+azOCFOECK-a3bBnRNr zefZ!H+8FSf%OX5+znV~NE5c&X`O*VI*v15f@xEo^J+&(LbEDA=Ze@{=#;A}rPXMbB z0T4d5!T+_2^Uk(Mu$G&h#J6j|{aswl(`)wxlB;C406&ddStvlA46?&to`OqezUzdJ z7tl7@pRV$|ZWt*cJjt2?Jxf(%BbfF*-x@*#6JP>wON>CdNDjm4Bm$E@{w=+}^%EGU zYn@*br@*Yg4D$+Js|3$5q%TTgESt`=l9R(Vcx8rycISBCZu^cv9 z>wmIOnwkg?o}fTR_+P4j8V4vyEf@;$=Q%k!(2Ulq z-&Wf^z=?Zwbd<@c7rL3_ItSpK3qNsjcpQL}hY~aT#^wHWk&(qrGy(6~&9S!PKrsi| zOd@Cjw-6Km%OhAkihI8si*&^F) zmNAe)0FVb1V@*R$WFT+R9VVPSF6=xUGI z=-APEqB^nTqN^w5E~i`gS27g6q1Qc28oS;1hX1spH%EUJJG}47+}`uLz3{a-pM0dJ zr;qbZ6l!8QKmrHMUc%pLdzjpIF@}&|Y_3dmJGpq`!Rm5$N2AR=4qNFPDkOiq3@`$a zAVh+I*Z9H57oZxOPv&vkE;oG#rf#0o?wBK7{ur>zWMItz`U0EF{xcwokOk!Ta@Ya% zoNXj7yOQ2P5&}@KW3^NL`UVEDR#%CF6;79%;Q*d4MH`jqfloXd|GQWe?s}VvLv3fk zLl3|mQD*?K>i6Oz?%}+h2*6ToEf!DM8=s3`ZiBLcg@W{dqEYT4OHe!7g^lBJl}qBAriPVy7*ZSs1v4et{eN)^FY~`&uSBT z^}U=dKa|Pf-~9Yil-lGpfKjd`=bx<~tCRnk?g;PUHZ$bx%+67EDxO%F=lrgydNhde zh%BTQejj5RmuxZeEhr^HL4@Jt->a*dpk*boZMnHd>Y0U&0H#>G89j5pvK<85r#nM>^WK3XRhX4~Z)vtT@njO$(LYa+Q4R24T9n0UN=7k{X87(y!H@S?Er+qp( z#Cumt+Q8+!hwAP9w6(YwdlJ83Hk1dT`uYM{b=2&I+OT?aML0b@BKz|V-DV0NXZl9F zR)eKE<0jW*#h~ry`vk$0RhPb2x9r~fPmmyqM*;WCekiZotl&R=yc_lH?Sh-Xzr%64 zT!3bV?l?!1jJ{`cd7uJPln@}s^m?g$gGbQm_C{k)B7 zFtp*b&@>6FS$nW~fFVw&@A}0Sf{(9lvS8YINJ(vUS1zpy*RIXG_oY%{uIhW>Qnz&% zLz$KbG{8)X70QGY@1*8zs+ZwMcoZ)6GCftsYRI$xg^qZ}>aG8~u-ugc0f&N%%Mf;H zR;F$B32ig#j!_o3gmgV-y(igt8y3z)qB0PTvcm9cFG-{2?wzv>yTfS~ZGV(5&Y!eb zf~>48jXHM}8r5H+ge$GnSx1*Vb@ zVD!2Hk08j)+brF!tP*Mxfyr)Q^Nn(Gd3jY5v#gekWfS1c31m_i5JbbbP;7ehL|d7? zP7JP&=Y?vR@C=gfxrVg-$Q#9idzk`FRI|CAbWvMxQ<+K*iBc$mWQJqh@7y;-ZI!Qi zbHkZ=wx7V5T#g;gVRL9OSLREl!^Id)&`z_MnZ-BofPgkxL9m@77wBduXz=|na-EGj z)+H8v-?nrkvyB*0osBjYtLaV8rI}q`yO{OFnLyv80J-K|rJm@spX8vfPDZ|soP3r< zo-gKf(T^OR=9*MyW01)#)X2Dz5+1k|_x?zrwEO+p$y4(8$-iz7D3h^xlvdYF^t`LR zb*GtfZ3v5Ln&jZP-y9ZNvPWoHghUFtLMY85-$kc#wXUZC0f=nw@mBYDUqqw*v@c#; zyC6(geWM#Fusy(GX+AhJvpb>#JoD0bnY%FWIFb44TeKk9!Q<`+rz3l9o8WL1Q93G z+h0s7JHDK3K%P6LLEsa#+4yzg=lq>^d;&Bpn9 zcTRU1VKONQ`P{>*&g>0c#3>wq;OCgMNB*1Ngt<6bZQiUi`HP5hIbS+2gFJU1Z9|Ft za(pXEZ!S5TCrVo7dQ9AQ^YnRMY8M_MW>11@GBHTQf;@prd}sbp3VMJa)e{%Bh-)o| zh}ZCZgZ%8`Lg?lFCz0;xL2ZCkX7awX*J-kc@VYrdgAuS@>mU#F87I>v=`V&nf4yOb zPxxi;dJ|;JLlB8Kk3cHs8okgTeN4PlFOAL^s@<3+fX5cXQ7LUS9%};} zg)GxR)M1SM>kF%-!aT9tNk$$$i^;KCrJ|Ijenbe8zQk@*;P%LE*Mj@ILOD$!GcUCF zC9zp3!GP{-de-y}qc|*8AoMB)VRu=~C^?Uoyda9?bcM(DWJSFkzbX#CP%wJ@m@4Q) z=B~C~;X$FylH|8HR{}|PO`NeKQWRucRCKvHs(mBdDi1ic?gw$2boq~8KgQEl1iD13cmPZUU{?* z=C0LRhd{^c9SB%Q;rB;_6Y;dD2Z!?~#9lXadvonDHB!l!QFt*5O;x!I>KE%hSZyAc zDBngf4zA+O7r~-GlAR|Vt1M@wLRlR8P6RCHsqxA$QE}AO8C5D4K0G`?+B-Tv&>FrF zTxHGtQbm)`;_LsVxmfE z(=?87X!?jm4SoK;wQ;24a(uo|8X}(~7){~+R85S+10?`uNY^Ov6Yn>0`ln9q#IO$F zOO?;yV$f{)%RJkZoyuyuG4ObsUF|-JIP~xu1c=MwhDSmE@y!6|q+N4U3KvqhV9+I} z`G#F67y@J`8`3yzU>O-_;Wtah&}me#i{!G!Dl#T_4_-fMrOQI7(y0CVxHQt>XydF= zrHuLZ;9(I4Xka~n#>q*ix)yC*QMKdF_p5)WAdAVMenTQ?6aC+KPxfQ^`30i?A-J)bvjpiObIEoaF!pA*y`lf4?6ACLzdz zV!G?oDyn&nzl7*d}9UvjMRDFy{riq~A zRt5!XIc^+phZV^ahTS?}AF%JuS7tAx3jWW_07F=~8J2Ta zJP@2+O$PeTIaa;egw7o?@~v^MuD3}y#?#dq|Jm-Yh#Ud z_kKHMa=X+^J>i00N>Fp(yb*%;%ND4DtK~TEJCgNc^4-^3Zgbblh%ubO0HXwzfI3{< ziK)r31-AOS-~wwqC&)fpwJz!10r&MQ=>2Pie-!*JtDO<}9+&I2ShOzu1-lm^CHEV{w;PbQI0$4M_Qk&s% zkv95S^-z!x*S{x^@kl_!cDg(vW=I4M)&mqrT0BS`bIC@!++SnN5RZg%Uey&T zju^90EI6q*{=UpgGq?Mxr^@5S%rj<(-HflE+Q0nMTki0N7@>?3QSk2!4QjAhbsi;L zMk~6p#cHm{5mWt-kuLX6hS^4sG28h3F4ugr9xGDaCIAGYz>Fdw`*@!@KzpReLG++m zo3~UPR>f-@EPuKde(B?R3z`1Mbp7R;Ht&g^5B_nU4o$N|O%4YkAGLtVX$7%vZ%#YZ zNyD5<==}!=$2CN*mFBRnX;~CZRQPE_AB9#;+WmS?(m*{5{FP=&VAcFr@G>I?5RDyU z8c>!t9E8t=QNNXzA5QpP$e>_lC>b;y@gwuJ?4KQV>7KWdY}4e)l_JI?ABNxkGa*1j zjKAhL*r*G=pxvdUQfk4UDxCdt4x6Mw30KC~B=5W6%C4a|Y5a>8B{(-}{0<$;k_p z5NdhZ^t=+5oe$oLB0TUdI9IWa_KHod0$rElw~Bd6s^0FGTBF+Y=2NK8MWaHM=M;k$ zsrnA@!v=SyQv~<}0^e`P2TT~uYFqX*NrAJTBhaU4AG0Z>95b zR(VPluKXHhn&DHkmSK+dVt?$7#yLzzGS%VJTbud5e%~OEWA*z=Jho+ib5K&tb)!D4 zRv9HdfCv+j`8lMPwkRgd2c#(ab_))zTb!Vjv6yP3VRH>zrh%icD-$*1ekNma$^s-b z?6{4;<5rdwbWtty49=CR8tcE#g2Ek3hG*B^{~#`z2zSBT z0PEd6hS-Rx!BC0mSFBzFoq7u*f~;&a1FH}A#%`@Fxbbtg}Ff*7}`OSl}EoXi26L_WM|GY zCe;)YqnFqiU;B6vVj1!YU!)T#zV@XVqN0NH$bXys!u(AK&8gBPxrBs;{zr-UA})^i zgVn4H5Yz4N@8Y1xEhu~^9?gw?6pjEZnXG5FORqrB6Woz=KNy<1@Q?x)Ola2HPez1| z*0Z=1tb-(tOs~T!F{|$OhlTdG+I8m4V$t~2jlD>&JktM*SWr;|6!XQpx;OlFddNO* zeFhAGestOm@M)b69T9MJ3nkzT4!|PODj1%ZxpM8QR-=&04nbDK5|T>&PO&ULkF9vi zcz}Lovz&nj+S`bYzl8e#gIL1A@ayH}#eA~n(^xhyl8ucGK$89r$mB6BVg0S?OyEH3(Ti%sdm8esqRs$(sId^;Krg+%!8B$L$m9HiIo6>$-p+hbtbfP8zphP6HO z#onH&th}txHzfNbZ-7{;S=Aug$Lf22{iB!T@ubP7>qY;0&oM#iR=77{)_ zoeEVubKeGuI1-XH^)2sl2jT)qxgUJrSU7AKKJ&}aqCZf!T6RE4}>O{=Vi*?c+b8_{4e0*dwc`yJwb8xcS za$A6Mivf7X07mWvD`mnLU*GPoqtPGJfR5E(e7IPTr&g)=L3c&a?8WFmNZu00#|A7E z08X-q#8?vxh0&cfcB_}k0|hMe8H3xCm9w)m6~U^#)z<*PYpX9-Ixo(WX57B-``zx!gejArVCb#?$aI zXb+JTDn%iIjoY8jU4U)nGvHt(n=j;}l52@mBnD^-b*{NS_9ZMh*I|HkSLC&}vFVGF z5ElNxX}kOdYI3kHTU-t) zZUDDr22F2TPV*6<*yf4H5E>PG1O7Tdr(`Pm1PBlH-TnPyqpB%@_dPl~8s*SI87^DE zM+oqx2W`u!v3kV6M*G!Q2@47JI;kdEOcxm*CevdJ8j{FFMr~`&Zg#R+Y%))Dd-3Og^=!omSE}Sl`+z4uB6SL2 zhvT)Cu?dkkLUx{t%_C)u=WC2vT5y)&TQ+1`?I1 zWgqMy50#R-f_=q10sP@oNi;Ch?){NCjy9t}Rsj652--^{RH`k2u#Xc3G{-~qqn#Tk zFOUFd+Z%yJXg*ztaC5Sf@^hN&zlPn$piAwG(=FIJ-BaeYGa6@Qykqw%9aU;;C_x*Z zHJiT^akexCtp7T`EE6~%aG?PHg1!Vwx$Y6clK-@mr4U;D*VotVH~WqEaX%=kd-}FGFFH!A ztaiRedbwpzmU&!LsnNAVVU4W`ZBnSdaQ9kyyFXkZ(5UV)JlnnX=K1*sLfvn%R8!oe z5Ejk`RPiR~{di<6F`wLY=TfzDk=PVK-tV{4P)Yi@)a(9XEUP)}5^L_W6$gJa2anyF zZ+GhCh{N95d!Xn6418DBdZt35%-3cyh?OkUv1OWe{V&l+4^2k5?;a6Hzb@8I@(Q{t zX)n!f+Mk`F0&I$A#i#D>j$Gdrl&ba`f&N``4!Q-fMY=oy$e`$eEi|4kWncp&x;2s_ z-x9WyR2K2`vAPtOOn_|w7Za0$&o05y{9HeG@*OPw!$W-$Re}PrkwVwq8Ql52`0)mn z+FsapYu^6X)#QWmqH#k~v!9j#Up?U7pK@!74qZ-;tuiR0)2LE~exRcIFb=3QBHl+w z+>W8o44JK;Q=Ic{CRRgi!YZ$71MV z(f9>m0yST1C`UeT%ApBp_gnjP{=@lT`5nPxx`f)za44}EmYLgVb|c&ignk@#0z`|e zRnYNLG)&ja%`vYUouL2WJkeKa=?~k(7+t06dh!3{CBcj@w|22YQ;r3K;Ai#qc85fhz<-POIH4QH|OC80Qop~TyFJs>cyL%zzpL|I;}`BFY#2qMzTISWO&xMcWq)TC@VV%@J6{3jnd&{P5d}wA zA-OXk4TU7lq5uv=RXTNs^I7l)1LmMAd2+Wr`ds2PGtAOa;zkQ?n^pG2@&Ul{?9<4nf<2LTESYf|iT zc5|Szf7v6@z{i=I!#aZsrlj1Q)b*Gr!rvcRLz6KmxPY2QjSe)Daxz@5m^(I#zPUPe zYJgGr!@8H|(t$bL#e~PXIEwAHpewG&)$xyfTf0wz(+}f6Es|WG);J9|*&tQg+nzqr zID_rtdIErJ`>xws1TZA8z+j^|pH5;YhhQ4a9>DCa`tVrJG63sV#?ZqaBAMu)IRmf<%v8|CpN1&w&A%BA1|2?*V#akS^R*_-Byk$|E_Dq|DUx2%2-w0yDs1_DkD_&&YXQ0>@UGMI z${Mu7D<9s_>ZYSut)z&JvyE!Q!=r`G_Y(+*9DZjwvoSB+Vag&NXGxGzAKYNMqQ~a7 zBI?2{HcMTg$vq@#z8;VFJ?iH$9e-~T=RqlUZ8rE)b?txGhe&|XV`Zg%bubIh?YI{) zKQ}$yr!ge8+IX8`&ZydwL-w;kB%LaS2DmJ|=-&w@yhzv`;d;wHQEufz=R4xC>or8B zm%DR<*4rb@d=kW)8{m4wo4(Hv*XB!o5m;3o(-eT=6_~b3XasGE@u!(!!Jp>cA4A2D zj4d!gCDCumy$WwKWTHkwO;jWmH@w&y+Av+K`6gXmCY8C+y32-7kU7z@LRL+$-Gau! z(fDt3lQL{e8++#uF;U59x0BHeLXSr%$;`>@7pN=U@uWkuR@54uff(&z%~~jfg{lT@ zrYiQxqKc5AAmk<^bq-y%4ZRE#-hIQau78WrvU#Nzt#g8Xga!LBbY2XQfUy+_nP4WJ`o1mO>I&{n*z66x$B%$NG3(tFME zqH5ZG9w7mbd7nSThakI;2@gPdfhZo3Qq*V-THP9gTE_CGV7nL!1M#3Y+M zOPQNDk;!z$=j>o?W77W$E6}G^KN7z_C=gjWHJ|z!j{Gl^cYKijOBUbX^MhI5=AZxi zF6ZOx%V<+ji5!!U08c!&YOC)vq*x?A43?Q0u=zu{j4ZV6`slB+iFvEbA;;(_rjWL0 zMSVR6AO+KSlcvsmf^%g-16=vGb9G`V8KLeKJl9|5W|x}_jk1$ZOGXgxw!-ktXNpHJT_4P4NlZYLIS(NS^(EjcFKve5U8&H!E}huE!V3`r{ss(hVdv7t1A-)l@nV`fXDVR8Ar zZYu}o_Rk=?>y5q$ z-cl97!O*96X+=+HL;K`%jc|@HxXd;A&N4IW#+Ki z_9pY0ZKX+g0cp&Tdz^EHxXjsxKo*@L1B=_qR5ZUoDdycBpK11qN`1IG3vINM%C>^F9DWd@@@V7q*-P{>C9{R2n_}T%7Oxo;W-0>aH=~)* zZ6?x-mzjZ*2^2&1 zl5hn|IxWs68jd0;vnn{-^QyB&2M7kf!bJ%BB0XcF{P|(`jR~hRHUGS$HQGwn>fLdHJ_~E%L!&@26C0x z%C*2XU2P$!%R;m?4nhWj5K8%sf>W_4&>FQjU0=K#;0TaS=ZHA6sqO;s%s=9D6OfQK zD!CMe{E_MU0;?|{;OZosgR+uR%WY^#9=7G-Su~0$kyYiOr3 z$yB*_M*yEsO_=ZjH4fOH1V@}MXa|UgouRo$ch)9x@jY1l{WD^R`4$l+{1=6M@){a5 zn=e;wYc|79{k!;nwe238AEukba}PJjrM{?hzI6HSaY{+Q52IDD2ooA0PY<_=%O(u2blkb7Oo$*%2l+1M28#DXUd&Ig>{2@G|o}M@byg|!@y#)JX5BIDt z-a-Xyi5vj_4*>dy;i&V?v$&qUwMwF2^TppUcgIHRCk^4i5=!PMI)jeJmRm!0a}c0N z57tXU(c}H#^B@^kr}{q9a2P}ipTKmrZxfzB$AQi4h>ppqMK&+C3*=3ptz-pSEQ*^~ z5hECm71*+}vY$?4{eVekON*idkWnPu-GQBzIu&7pyJMLc>k+hXDvDs>)D!GmQ^fm2 z(zZ*lei)zCOinysjNQP+IalY=Ymc_peNr!x!I(@lntf|AmD~3ye6Uof#PO>4*(5CO z&svzK6XYB=Oup+VW`j!Yvy^+zTnAM)y@aJxQF%ZiakM@^PL;|$_UtzT9 zN=_rmJ=je4$fN4z@mBfyGy&y7$=Dr^ZtDB>>pMH8_IDG+SNlsyfl#l`@0HojVbouq z?q7!?n1LNJMRFQd2|s=!TXS-9Ds052CPzL;7<77E!t{U`k`o_!%VOv;vl^?borm<$ z>9xMNufM?lA>y#(LF7ouul(wJzJWjysUQ4DTgDy`%WPLXBMU~vM#>d&_{@ocW-3s* zNL{q-FL_y+_4?7)_kGrfL3YWA7T5+JCXZng>*cDio*ChqtY*XYB~W1DB*w4s{qnM2 z*w}BtVFY#{(QVxsIQ-)SA=t%Oa}OS}A52igeC+$?YHu&3pirz+>aTZuIR9Cv-IFLQ zi^$iD;B?JLERMvZAUzVS+)=LB=RwVdjy5^?FnMg?27!7zl7CbjpXaAEd0fVYx|MI; zJ25(x6A5?+mzIWToJtQUfv8*51Uo-FWW!i7WiXYQAdRD(g6^o+j6md}$Magkzf&SvpCUuh?L&(7 z{E*oz3_^N;)emUkJg6&nNfZy)_#6|Kmq;$c&(9}YhROCz778_Rxs4i zCb+li0@he# zx`V0fw^kRgm~S#8fF0Dk3e&HF!4+{O3J7on_Bf+^c80l%@Oj=TpqAVk2^x#>JMT>p z0v7MSBwF=?^p~8YT>;#pD;M7$$!Y5YOdvQ#It4D7^-)LoJbWn} zQ@M@dXw(_Ld*3^T#>S2$zrw>`3=I$w@XGXkYc0m)2&+GzL7-BaQSi)MUZCW+n@%nP z?0OAkmPvW^yvneyz((b>yJ|eK7@hEGGj+)gH(}_8p*lVDxgZ|Qh_65L6J^qxAOW}$ z_i-4JfOjsz)>l-7AI3Ge+oNFq7u_(W+;T5%BPTY0VLlZv|G9tjy`J?VpKKyHFkZmE z`KfzTX*ENwI!m1oqEQ!boW%MyVu<`-smxraNNi>$lXYq6|L~MUGCCiZy1KqTNdQ{} zBLS08Pp=8?Vx0x45ZVia*0dEPqe`V3hR4ND(#OVZGP%rH@(3Z7%?)R5i z4Mzlo5x>BYT_~VC5=xPqCNs=rfax8dh=BX!nr|XWx_4OA(-+~!JbzUr?rvZ}rB#nj zWFzr(K`yStb?W~~3eIhbx~a7@rS~p)F#E|0{%|lrwoKRQH$uz~!KMYKO@ z(S6OAJBp6fjH#V)^#x=7e_1&vi|pXgX$zrUqylvFOuC>;vpzmX%|pQx%!wr`|1wBn z)Yp4p5D47g`kKo8Ys9Wod35b-<2=ISn88+E(qJ+@FCo9@N8Ob>N?b!g6R(>7k#@Ga zgy-{kgJGnbFD46QGucFH)9Y;8vKS&k%r4%F1zi+P-$WwC99ESqPjI{%_w zUgqqlQjLM$GnC_Vz;L9~ej$+vSb%``E5R92lm#{*=x9|JYv>{d7m1gB^Q7YLC~b~m zz$o*}#N$HESJi4FWNiO`|FXxT;n9FSb+hkKo9{1oyG9P{p@}RkEW!X2^>Ddgq2_d9 zF+d=RH2n1YPG_N|ywad+1LjX+0mvHQ<29b=_o|l4>%|Iq@G09W8m+le;y9Sa5q*ZH zrh9h&Q1@Rl#rmIyTZ4VnMPC4Kp?t>&iEV7kt?Hsj_j>6vCN#*98b$vG-n-| z556i`OU)%wg$ll+im&wjm(R4Xo?~nDUM8tczt_mv{{a?iE-GLvD=GQ@ovv$E@5ZH> zpiJtF>`#uTq&XYQT>JH9Dm@Y+z(__Jwyp<{*MbcBfK%nTjjbpzi76ohXd{!Ar#eT= z9;lZ)NL@K!%eQNvg7`)ok5h>Q3a<{BKU6NS!G=_pFBNXyv&NWno+1()63b^5|1G${ z8vNe=xSQ|$AsEVG#q%3r`{2Y(JMNw6f{W#jAWUY!qh6gZ{PRnKASat3&zLP5mbLgfNO&u4;Rl0{-O;v6+GEV$ zjnbKFk|SH*9_K$_t6ymuZ0#mp2l<+>5>Dd_=k~amH}GK~_MKN4l#%ftcE_N4$ROz! zEZ>ns!@$Ff02VL>z?0m}VVILT*;_Q;QLV-NF-&Et!DUNHMWUB_n^+Mb&B}B+VOm|s z#RlZA6|`uu7|K%d0M_aEpdx{RVe=Q%!!BUAJ=|h_jAb=#?mMaxwaBino(LvaYqOuy zMf4T9+P7kp-9`ToS8p9vRU7qdtCX}flF}e8-O|#bG}0yA9n#(1UDDm%-6192-OYFN zyyrdR8)ppvatvVa#bWJy&iT8p$!ekTGvKidbe)k{jANoCzQ%s;sj!;*A=WQ7(dH2x zaA(R%XS1E|B{fuYzV*QIPn3#~s;?@FCaRvzMDZtGPgwXE*4eka;P0I1F*}(>FuWcL zo2iPZ=%~0FaEr^>#&luPI!~#}#?6}A_jkW?q<;)S z&sZLMwWw#W&O^ae|5B&b;p|;LfJjMm3UVXcIXwPEGg*EOD^|SBy|TI*IOKSKiSY>G zeRiCkb#YNP&M|w!Fo!x?ITlz3~FgLW^ zeeBng(ezgYp$y^z5ei>bN3!?{DiBlV1 z4e^#A%^DTf>?ay?vm}uO6?v7X=UY%Gcb{@uytGCt=f|6mLO>(<^t=srM4AG=L;+lYq^WmvCAo`=|TFYJ|)XD z*x_=CN4dC)C8|&@?4!$nm^RzzN~C@sR@I_xrTUNQe#P>&5ENkS)|2v~09N3De_nPM1wN0N(ZXlOzijAB4>tE`mc4TOULs^mZkXOt+%PT>(9|x2{o>4+r_*- z-=k`z(pCxi?ugY4+y*#b67{JaSRpn?&u)sCu>Uha5NE8N7OPYi9Eu$D#UdF(M_KmS zhWs&&QH+G7shKDjQCYIJ#TYeIvf#*+8~I{ABttybIT7I|`>kI6o+%kCEbPP9AN?2e zc@0rDcwHEWAmUvInXU?vt@R7VwJHJYA+ahI@lE2Nj z=WRkFZTId3{30a-tkA;bm@j>^{9*r(1t>l-*up7d*46%|g^$)jXtzr&57b@P2 z^9gwU%yNacMC{zIVa178+8#G3G;>Mq69b6wvjh>4En$|NO(&Odgl)HL{eHi|5S$du zss|*k;aQi*sRXAo1%ukr0X{MwGDCk@d5(g-ljYrSQ#sw;8HE^8LwOZ!}K$f=u$Y&*}(US9??Y`iWtkuFiW_ z4|c+b??0Yt;^lEEmxD9{)PH4stIZ7^4r>9udGQ7ojc8Zca!+W;ox?7L-Axdpq$rY* zrR*pB<0aoPe2m0a9f4)2yBUA#kb{ zP=gm!DPXh1G-ul5g&&8qSk1DSafVAl3%CcWF;HNaJR@RNP#cI`X>13_j$POP)5lflAKbF*JJV6AE-V&D^#q9jhB z#lOz8vk?;-94<(^q`IE=!m88BEk>HTvJYb}ju&vXw5-DZZ@b3r=CSATc;2&NQB3rb zB3Fk53r)E`OuQ#j_%dqC#Or~G`asC*)|2DQ1%5O(n}tL-M_1cR1Z%|*gE3%&qKhN0 zSadCAB==`Ib(-9_*$I})1<%f7<(d>+u=Pg5?vI)}KdB+r^EmmSxzuL=>;6TlnE3_J z>!B-Xd20h^FyI%ly4pLF&}@r;4zeZJ&36or3*2IlfA^?4I4qZtyc=14fPj&3)}(zq|sO$tc_|D{sM@T~4$eo+J zw%0&$%;Qd^5wHCN5z~}hC6nXuPVHJdu_rK~#NdL}(z@B3I?9q-*wy-m)v~;*1Aai& z-nuz7l^RR5TI%R80bMxZYSoR)?Ss1?e^|=-U!v_!cyxNI_=Ft4<_nqmC4r{AnTaMr zLDM>JtcrJ9t3Pbi?drozyLNO2@zhm*yWO~)-)(dJO<<|qA2uwl z$+M;@EOR8o>Gt-oO!dqGBgDpzMc;83R{2A}@bu9a;mSz}A>6J4IGXto@vM^UV2x}F z5tMT&!?G;?rqS`)nV#XA#jNV1R%_(Hh!+lgS9sIqDq>KIbuD=xFyN%_`~MZcli0~ABl=uJ;J zt*Hn|fEfIUBG#u4xZ4xfh1u9_#aCw3N3Yx&2zC_?%GM8euQzi*Q3GMb+v5^thAcwW z8v;n$fzKyg+MKoYsW8D;)Y>(6JgmcMUg9oHh2mhfEt8`~(5)mubTT75Em-JqmkSwrF=And(Kcg=$-k7|({IVNFg-wPt30q@V z-BUAuN1RSXSJ4SRJi)7siuH9V&M3m-sr|C7>_4UwLXb^P{1bFhu$f=8d6*OPl$SeR zKmFxO0qhkb2uB5A{kFpqeGI&%jzzD}$sR2nn&(yIik--QZsMcXRl)pe~B z(3acnbqAR3oQsXm%=YRk$=nqrOw5Q=pQk&RD+ouEG)-?gvPevc-BaPM;B)LPzXN9HNh^|3z+{d|Gfau{=2RTO_YBTV_9bZGg;sgs;^ zQ6%P0*-{7BiOe@@m4+DHhlQg)o%tr4;3RA=2a?Bkf~yBuK83-x*;=jYa>bho;}N4y zqsO)177Iz$wnHd2|9sdk;|a)!U}(O%3yNf#)%o#{HRaGx4)->BkU`SxV$0->$fY@v zwZ@D1(0!~g2@dVlm(+3GeKYetY3514q2^v zd&ic4YqzLn3QiekmHeaMp;qdAA3KB|1yqPQufgf)RlUgWI;96_M%AuH&Lr&FD&PL^ zKiM^rBys;D#tx!m^wpZFV^sBBVKx-}^2znc_bXrEhWBS1j_9s&?q0I~7k1T?jq0m~ z_2I#H(?%Zuh#HE^Ea`=z+0X`WkrV4dNk%wYb&I&P%mPL@NQF#a0w)tSA6x$i!fVRy zeo0B0jaw;G{)ld0Ku0p9^&ydFFfrOhZQ6Mk8CVN2*5LCbadsNKcTJX`H`D_E2 zaeNjW=8mkVzD@8y#^JTthPCfF6kNqHV6PLdLo2rHL-q7Ue*VKHR7?)(TEkCf&|3YJ zP^L=u|4uY~va?A(|0wwVx5`i#;NKurG(N%xLP59ZTkz`Y>BJ7D>edpr#U&*Z`7#z2 z>0C-$JJIOWZad{F6`UO1QS|2Z|yyAQq)Cyil`f4CRgNVc24iMPW z<7FC+{O{iKa6uVc1)jVh&5RjDPZb+Z9=oNtTc`5S&u&w{_jHQzd%7LQS_ zKW+<#_R=yIx*YtVpTA5rU8ol0+%biOw2Ue95l~aZMI)E&j-mPrD{^}_N zLXeKhNmPFjl3S1H^GbDn9Um4dl$WJd_I?~{*=0A%d9R^eAzjI9Ff1Avq`>GP>#&<6 z5iWj(OQKfF&Hg!XB?Pgl`+}1G8ZZ&%cU~>ZYDd;5tkP?*I$!2ye@5BB@#PllRdBpGK;y! z9g6E48{TSl79^JrjPtpRy{SzOa$=|^k`JSNadN8*_f@Q{! z2Fk4la_w($9riXhXg6tqn19utfcZj4o-UvJV*ntN2TI_0J)RMGcv{}Ky(iZOVK4*v zMlBfexTLtSFm(v#gCz(%wVFO3>5&Sfe{v{2A&v*felt_OU8dm}BDKZAXTc+K^^#Do zV30uGve04MM8;fWS8l22Ar`@uY1T$WTre0$mPh%$BoDjw;_5DdvxF)wfp z!C{D(Xny+bF0V7HfY&?53LE(1%GTe20a>5-XDik59Z)uHb4`YNptrKN7WvoeW_h&0 z!YB|m4eUPI+FgG0O%8;BQXrWl$7g;gmu-Ludil@|sJlF&@M1*sz>I==Vgt;hIW|j} z|44`{wLGHT(?OiI=X`4@l0IDrJw(F<_)I@N zs2RIU&`pT_wcQy>B?7s>7>pj z#gxiJ=$lLnPguul>6!)VF0!qTZ?nkJbi-M5p8?1q+p@3{MMnyqR+AsqcE$l(p3MGa z1qutxP`%YvrBqDyl~D(lKb@CnHxL`N-k+vwc0GIdfQ%PmB2S)_l*I1-$gWyvPV4ii zWCZEylj!##wklRUP@q*IjR$HPy;YA}qNk@I>vDgg`?i;-N8`!I+LjR&Zrge~m8lxp zVpo0c77rN0X6{$OrM&r9fLqy}&L;H_bi@S*;W_P%O+#u~Kl0p$dLwV~ADdQrZQBQP zWdN3rc+C#0UEVOk0zNy~ajD6H-pkv&$6cPkvqt|wv7*gojt<0_OvCTxbH!mL+=?Fg zUf=?+f&G`!c31XtcS7*6N|^!))GATYQw)SFPW4^?Zi3Yznw_o$mTsvk`pPyLKNAOD zn6PWKgq+vx^9xGAt2XZWut;?s_>5G%9%8@1#nSmB07lPan|R-YOr}G?@w3X-nl@jF zMZQZN3VOe^qQ}q~THo5ylg8oVyt+?QQ`5Z6=ap8s;HfBIiDu`m*YS10Q_eb&57fI2 z@gFudv~=q)&im(lhXYQ4o`o6{-KXM#N^%S9b2^Ww`x^BjQs&oJoSm6VTWOhybO7-X zY)E==ONkkd8}#e*kvbwQ0`TCPfdRE-

}{y$;d%N1aJKw5RaXJ>yrPqzX?wy0CV{T)?LINC!P=rvZ?QEQkUw?}Xp z4c$&x`L%!I;^M;lvyiU=CIdBetqO~%il)3xK3l~kq`AWe z>+vI2qf{HwF0b`F4R#*Xqi7}5_VNaaoY8kLc+cHW56Gv$3;MonN5B_8*l9BdCyqRG zr9xNK2jT~axmrElvv9|jQc<0r-H~qF#)cUHR}`3**`J<1x;2i+d<7Q9p2tftB;LXa z7)$L=@+yqj$hUS}>h6XF$@gGLSK@+_hNSCVAfSH^D2nN!krk!1}f`x*H1eh_+9QH2~J0od*vhS-f6& zL{&U-y9987Se4$S?g>ZuZ4lYV8)Na$t-T;X-f5GBk!HDtgy`jrb{kBCyIo$ z!6fCcRO-XSq05r0X0mK6Dk=&T$3_?tvRl5{n|gU1VT6SML}S6nYF9r_N_%^Ioxf7@ zlLc}lyZif{djn`zg7j*62(LblxMxx*@IV&^;X`0f%A<^@QIV(dH=W9>p`>8Np#Je8 z8-6ZV8eXo+9^VQP5l3i*Aa8VhM;zs^4yx%NTxO$)^4i$$>RHaboiAsvx&LVd>tm{) zjn&HVDbq+w-TV^pj7uuTj#d>KY=sRDV{*l`c~}2{Ts6uh8hUW8#dc4AujNLxCn#1n zD@W&!sEZT|;0-BYN)#YkXIM&H@IIPh9LFjsO+=5C;X^nqlnOm~m|#5s{j|&^Ll*j` zAqZFNc~nR17dDVB@~nOP^>$CVxU+4#J@B92gLDl=#ko*2phv z|0bwRYwE~zUEt(z6-;nEjo&{&Hp^ywk`~u7OJF-IQFC^y{4yJ4WyEv6^jEs0GKS4T&z)9y_YYk zBNB+y&lsu^zc5e~A1BCSF@ef?{)OEu3_0glg!1v5e;%&>V}0S4R#w7VIqhF&ismAo z+fuBl&N7DXNdq^*6(U``M6q+qmF~*$PG5hB=l8w*{rxXBlmq5_sC;kVv-0z4i+A#$z|vj?;Z2j|MzEC; zfsl|NpshuUI}`;Z20;{zd{!5&`Y@CeU8PE7^p5ihtB$3678FYfS2t!;?|JQFep{`dD**7w(sfn)T&# zI#(|CU8ypp2Yg0MF@8u+=fVOUh!*hI7|(b#RfoTV z3INU?5buE)@`hPkUq=8SOttY}`BME?Q*DT<<@AE8BR>}Hp8yper*rgX$7ejvHg^nt zayu(P9_-49+JTe$WwtM_SC;YyN*IpP3e1yT= zO$(LzmLo|lUpts6g-E0CZ%@y>9douc3g6;D=Wm7JbJ_kriKhopIV<>xA^G2|uhl%G0aiBr#ArqKR2-2$rstvaz_*wfg6+DY2CjYz$zP~n^e&iR$b7xCo2$c_t&w2 z+aV$%a=cxA3raegxG$(ICbONbZVy={qT#Urn=SMpnJ1M78N8@4+IKE4Fh<7YW}Q$9 zpsQ>Suo`Kp-Tl0-`3hY|?M|OIq}BhS7NiZHMarpIt%1{d_27V7hqKOPPQzTxtz4@w z6kt6nm3qQ)=-lB&nURV&LiapYu|ueS2<2r zCDd+fMF0?Ph=^ky4t&qknH^5Fro8}NVJdig&hKdC(kTI1!ofiwLQ*;H+ku^F6B880 z84y0i6;36&U9vP9Mv643+ zpBx|m@D|;A^Bcb4rc8QcP)sB@>9fOWgyX)SfA_0g00cjvlHbF_cL9%exm%z8QnM9_ zg6cfD+56hySl#-J#3KKE(cMOfY;!xTKVwMaQ~)8P{$wtWD!>1IgCJW(b~$l(cNd5^ z5hKsY!eDTS4UWKGpb7xqf~lLp+56E%E*Q2_O{KlBJeM95S5bYJ9xsK*!vwGM??>Aers0B7* zSWEmLxJTLm{1TwSH@F2}2sk(Z@-=iXUH!E=kZlnWYw!)E-x*sL$q$Pb$_fKf&gQ9l!LI!hpPR%I>aGUOjVTKnt zaG+p#0gEC0wOBnUR$8A(=eiMKk7`ROHsx)S(kRfRM`ppRY{xXdk z)^WIqIF0Y7bJg<#tO!AD5a|GM*K(b^w`U1>-DEM8U5`kwEtY3kRBigPaBQyDrVmS+ zL=|L3U&u3aSSwwBJuDZiJKj35uh{h5!Y+^J%h&+D5Ut931y@gv@lV2FtfNk#QQAsM z2;5sJ2rIVSs-Y^AkW&o#`lzZ%(8VD3A42%z6C0aK8o@H&>h=?l&s`y)2vxMtaCcU} zwpvQ?#oYkJ{Q{Q+1eewlm|So<9g@OUcy_d|i;r~XxyYHBCo{xn~Giu|W ztyT9R!UL`3M1ht5yhbZ^orvLO0$l>99pW(Nq1WBM)wg8U&?Qu+5w#z}+r!TDbvIt> z7ttWt2KqfE^NnUW#l`Vjb7_h5Ef=H&eAZ}3`7+C2$I_%Kba!meIKc()TqQEIY`5lR zHA@TPhmg#@Q0iN8v&lWs46ri^f2`7qj&*;yHhbMD1I}WP2Qi6~`~Rc=xmbOM-yBPa zNc@tEW>d}M%-VJQGhb+NnX9xwwFdGW#{f;AmKG5KZ(j8fZD)*AYj>O##ipsoBwnO& z(iaG;>Kj_nW-BYK_*}29{PPS{U2j*ZRk9hm3Q%lU8uGl$%@*aFUGEf26)Z0cXhEi! z9M)MxPC8u!H@nZ?*Vk8IX(*OH1P%i}*e%O6k(g3h7Ui1FaC8{-*xr2pA**iUhroGD z7ibP1LfqF5M=NC9^|&m?0ifp30^BFSRv>{<6-%GXdhKgS{aaIe=$@T_ELe z(+$G$KL?QqdyrEwT4Pmj+73XDifDDZKx(iS9+6As@o#O0frNzSSGvh(5qp=+^Dt&l zrq?5rRE`6Jg<|+#BAt_I?AnM(stfHa8rma$K*qJl2=bAVY1KH|nS80!8ZND(&#oMs zQSV5yxo)*{SfwEI4W^QoXRw+Fac`PQxHjFb+WP?;ABNz zM1sv~YE)gK+13gIQ*n-A6%~vtt)UBVwdJv1A~3HTVRw zT_FVz_;v@ef@kYc6~Cq!8=(8+=_%6MPNO8?)N@~Y+CR0Zi*tY5^+zD^0oK`Ut7vi^ z*wiSnDka>6+DtyAHY);HxYohmjVof$x743eu6iY!sURJD-n+w$n5SL^$8F{(H@^9 z|Mo!e zQN|dbt>0}$y<4_bRj$=3F7N-P$K`S@Der{P8=^s!YJ&=Wtr-A|=8~m1@c1oW0~hGk zV$%zE8Db*#V;^t+y~wscj@_AMiO-;(Yb<4q`b0*eY>xt)M0k1zIK;{ucKSmrxxjW6jIJ zHi-RAF|a;D3L7W#M4_!SQ|$G_{0r$k=qq>@6;U@gH#1nQa5s=OYE7wKA4nska2V%a1*E!DzrFMZE570tX0w@H=;ZXpse_lJU$RUl`JD1q>7x1zBe30nfV9?(+{&XzsP5le%e+W&xiO=(7LMvEqw(2iU1Z zhr?zb0s<3Rk{^9_CGy3&h0{^yp(Cx+dbq#O0tMOW z<>l{8ZMx|qO^yAE3rR88+p{MaJadk&zk9y7VM~p6YR(pjTeG_bL0SOMzy#j|O)(`b zj1@@YH!-(0VUghCjtPl$j_*%VsHaQ5i<*OAgFBPa_$NGWqd>r$cTHdBTb=!PSj05_ zn5zY31)BR99wGqpUIWHdccV7dd9Y4wcd4ODnWV6&C>T6T18pv2i+|Wvl~Wxk0Ldjd z8o{#R+Xpy(BY%#-vVCgoTHD|EYp{_52VckrGEPv$*;2DRA5t2OCZKG%e!j3kk8v~C_@TC45+$~DHZ z>bUp{;8?k-x3_nPQy`{|60vn>o_%3Dc3Yk=3E zbjXAMAA44Xp@i;#sQ%F7=lRRp%Y!)?YQi=~lqno6eW>PaF21S`0i#tCcI2 zeCY#UK2act_!>KBlb!VG4eiY^UxrIya>!7}Ws-!zViJ z?c744E*!Mxjz7x7tU=iyaR(}FZ*0yMMEEYZx{(Yd{R`xI*cMOECT`v=!@LgO(U=Es z`0?@Gc!3qmo<}Jq&3SgzxKg1iDi-|`s)@yr5+$))SR=c`ECGmXi*AtHnvyB2&aaN9 zQWOCH-?UiKjpz=yU3K}d`<>gfjq(OP@9_v(uwlir`6OUxEG^S95S3D)GDf|~IR`8O zu-l*J??b}(ZIck_zr@ep-| zOfC(@Th(ZD56xKB0yDVHD=wW|hF4*SBelc9EIzOy3V6PQTBrFB5!sFCDKBS^6`Dc2 zkgAg)=1_x-J+-{)&ESV;Fq!=tN!cNQT3@7u&!{6rQEmQRFo%vb@?g1%O2g%Dwfz}> z(&g@5t^L7Pm8kyU=GEjwqDG@RGL1SjUW2uMoLZKE7muEisO$NiS1Ol`_SlLF@szw? zhrO+97>Ley={XZddRblMZA3-vbqbrFjabQ0XT}6qnBxmNr*+Z*G`F2yUz9$2;h&r(RQy~mr9VHg z(R#~7_9lHvBTYXFmp@#x_Wao3CCJ}fFBF21`QtI;+s>+BLaGCCma<<#9d}{E`GV#r@8-lIZYmzRNJKYNAyT4KUtv@6xQI<`8G}cvn z#H3|z?wiQu5?CG=Dfb_olJmm&jlFVWygoZS_`@O9cbA|>rX58A!HxX7<0XzoGJuYa zrnGvdwIqq@lqyq8dHpPQbl>AC{Pc>Z_{^DDAmFYj#-ye>ew(XQLVfs^g@+676JXQ! zYI>liN##IAgQ*FvxESn4ydOE_a>r;99gB8{P&wT^iw?L&JoAL&vg?#KG=_xb8sP_) z?2nwBjo57F&EGzqp6B9%2H=MSdlgiNBsDgMewx6i_j@p|qC)EXG*7D}V}>t*Tn=f( z7AvMv5egzRe{#E7&0QS;O(m$OI5|0u9%rK0x%oxq68MQI69pw(df;~~#MqL=8NLkq zf|ZUyUn&e`Km$k$pozC|F_m ziv%DdAGWB`V~cl_&nGt9(fuT_ECQd1h0nivo#6EprTBmeyzR+Vi1SAvTki`%@mVK3 zClU=Cj@mW(W|y^J_oGPhox-1(&R#^tEJ+>pY9n5&Q1MoG#O~BZI!dC{fIhb}|D>0{=%_H}9fELI!aE&1~vk3s794j>)RXu5#Y5M%i`WD%;Z z;{_7>pVm}gTV2g@Tn^@IZ7OKn1+wmZUpZ z`W>6O9+8sp{9GTV+~Xfd2wsIcs#VXDO>ao!?s5|cG!jFX=hIGwPJ^%B9v6`btFNeo z-u^5t^>(S@=sbeN5d5T5T;^9#^{9HxZojKp6oF^1Je->tP6)n|r(aVKI#D4ca9ICPhLMQYK*9S3676%^4M`kzmAiL2PGEjC zGn2wAXBMv7W;uOYbdXG1l0aSJWk{UGHCvnsGtf__#S8nQ57$oyWgCW;AE({jf1M>s zu#FjaA#^W{UpV4AB4oRp5pcl$%bE?HCPDBJ%{FzWh+3W>6%i*!^o9U&3)MGinCwZj zYfJu+^56H`a`b8w4H$oE&90kogsZ^#WwklXVr#4-Q~I_ucWi0)8KqVwr9%NMf#kP} zacp^k9Ew)5+&n~dJ%f)i$wtM&M0$W3Lm`&waBRNn&^VA_cR#~?=k0+c_PsBh6J-%O ztxKve-{TRHLDISlTs8Fu9SW~>bQ&Wc(4`xwf%TSuB;>X2D7*6|$n?Adp4Gal1Qzf-!@CyQo7)H|l>lsRR|y`|(aUomvTl-U8>bH^6|ZyH}w_Y4{E z|9x+k$u_VMx?#Pt5<`_V28GTiMRUHe*r|tLBn??hY#b0kG4YSndS38m_%cj~uc?$` zQ#YfJy)^{OS7@{OxFk3|g?D-DdE}Fg#lc;R!|)pH;@_4G+RMnsG7r>!uru{Z+lX5o zmi}%ukjm27`!vg1UZYMk2-|aTlO9=Tjg=#jP9QCl(({0)IsfXV1c3Nss1`CVD*}-G z_oOuBCEO+#%e)K%U%9U!AO*r-1#P6WV6mmp!!^D;7fo6cq|4;$wGq!adcMOjis0PH z{b=`$cNBIR7bRqqRM(&m#?2!pVYAp^e?hpQ>&$)7~ zLT?#ctQ0G=!ZphKyeY=yhAHW5=|7UJSd7&XPUOVHHdYsqX+o_MqQfcerRd(*^<6cM zCUBKDfl&6{YA`|eFlZdczUHRxa67=q5(5nlQv0we5h#&p=g4VJT3W2eCsFA&nm>4| z{D=i>PS!xr(P$|rkd(Ip_^`atV1 zjx|n-kOpsIyKDu1=xU15ih!q%j+3kMfxUJ!bV#tM^V>;A zW-p(GRJocGeb1^g?POyhQa2l6#ktjo_IqvVLwHx zoMXYt(c8CXs~0_lCD6mYo-DonaVp4C55wJmf98FocpzPJS@*k8I?l2g;!I$$Kr3b& z2{;X8w?AW5H0KxwnO>S#?Vn_nd|m<3688$Ri=rFRY$TNiCA~digO5L5nI!-WB$d>@ zimKrv|6G=9GFBm~yh&^BhA?C+UNP|fgW>LcfyS)VV!QB((Kd~B#AA277}^iJte0fZ zmF=WKr}D2NK*aE&&JBc151}XTkOPR$zt=iwJ53_(F+Dy^iu#F}&UZ>1{OIWf9vTuP zJoeuYQCBzw1mwVv%o7=U?bG`ehx{HS3?>?3m(b6lHsB9N}gp zM0T2_Z;zLZ{oPn7fua1F#n4#uhqkoHehW2=CsPYPS&R~R2RftEjm^i zWtMX)w-cP*!RKu~T`*mH6%dl5{|}r}o4w%%nlSJn8CCCL&J=`On732Y$nVLyfXLVr zcriDgEh_Vx{xc6m0^SC&z(|a8evc-0bNI$KC-mZGbqgHh^u3DsU6zt+tNkXG$X#UI~x3>0i%0NP5LAheQ zW4>4kzRWw_0~@~YuH=w_E~|GM6Q*Nq6G(r0-l(;7T>U&JwBC@kZb4->3mv7Z(9p> zn7;JOH$BNvjQ#R0`PxuEw0^SfJBi*XdfR2Z?S}jDvoy^dY-}mhmyXP}`kUK^_fB^v zdE|rxm2DTNh10)o6Wfvl@xNa@B^cN8sP=^p_6FN)tt3HMOC91h?Y^m-BkX=|r`mSB z8IU<_YP;&FX<~GMUJ*+zM0AlE%1s~Fr}=A(-9ZFPHQ#Pz{S_Jf zev~?intGJ*Ox<_g2IyT@16W=moE7)Yb5s(8;&=(tH0@o^Nrbu+9+9TEOjZ}AXuXN% zUhxjL!3P$Z!&1v4^l>y#g7YS3-rG+GuxI4Q-0p#_5IgIqdqv7E8q6vY>Rr z?a7N4x4zqei_JWPw^R1)6!WPjyD;-D{<5v5`^RLAM9O<8}B-~MfI|L^^DUQ!e zeBYJ*{6%Zb_gb+Ds06R*2gSUp_}CeSQk>L*v&HtD^k zD*j{F1b~9np!sIY>Hzne0aJuLI(cQ!Ay&fP?Vb%S)?7YT8f;(ghRV*f0eJS7)6NEJ z2P@Mdn&I>%W@q= z_+G>+og@aO{>vjSyNsRaqB%&hUEVv&Rb1{X7fR+FnLTg5S0D2U?$?iC%<&7se@nBs*X6nKmf}em))s6zvR`QiyW0Q;@D1uAP zWeJ8$Zr#u=T+BgLXlkaiuDf0Nr2$9NxoRl0(rfRWCUO;)Db?~Z?IO0@UWEKt*Jqk0 zpI?a6TwW@QK4{`hlAh(2C(!e>^x!oP*&yG#U!gfgHq{kUbgH7-(y1noJ|;TPhzF+3 zh<9XD1u)gtO=P)~$&vGlrIT@Bifj*jcVM*3)O%`1DNt*9cU_k58%Ap!{5YrSiqW;f zcy|5N7ci#W5IO#E2NQH1p8mw(vPE#H!uU4dA$hGmiAuS=SAY~zrRFICfWkh}i#zLw zrB40HOv$^hV5yUOImU9%N$1DH=iR(tlP?&=t`ykh4X+7wANSN`ow>}5bpyAfYqULb zWexMdk2vH{YW=7$k29L4nq*QTD$Wb%F3iw9don6kT;k7ubEg{&|uxw#M*>=Gr-hIqg-&4P2(;F-H?^sp~C~;YI z?%aPui}^i3>0v%yCP%d10MW<_4!y1GA|4Cd(28ROUMOy=ndY&mok1i)n1lYJbmlsm zfm2M#PT_ZG=tYa?^{M5u4tU%YI-V~`oOE@J(UrUkmgGWHLzlxEqBCKhrYw}(98VGY=-A&ErQR_K!*6CAMz>;P0dZu1dVexCSJHG5&KdUtw4H_T5=W)h`6F!tS0)7}?oHn=FRd(P8!@Un zc}qx?5zKpkCbG+y@t0nP0jPhz2_7O$W;h7@v&@}vdsCT2k|Oqk6OTawznEi|8ib`P z`1Fk^z#n;rUE7{=#a6lP#!+BWtVD3FeP|JJOKB&uCVOs~9wJXKpj)yaA{gC<1tWY( ziHy{XQq=EZM^uoHXC)8W3KcLnVC=f-r^Dx?xK5YSc=bElF=%cq>vzGLwv6@=7-7gjRk6TXbLAfDXoE7vz zSgb<%+bo&s{G!t@;*8l#Mt?IZ!6o@QwCi5d!!kbGY!0Pd%+fo6f5Bj73DiG17*|(C zvArEvQIY+dErNzKWOA1IWu8o_ne|2-bHKmxXMWbkDU{>Q8dGB}8H{j7fZnpfUQ$Uh%8Q+fs zk-_W-n=Ljc;mE`2@3Zw(Wlp#Ehxk8*%JD8#CLLDK&6p3k(D40|i8PN*I}^|O8kPOT zpxN(4K|*qX4>*M<8tte7#QWYbnHFVzs5@O@v6#XA7<#%$37JAV1vCb_2KeMzQy#s= zC~uF4!HNL}<2SE4mj7qhKp?c!;!6DiIvM0I`L+9V76Ab?x-*Eb|9S-@q&RHHjT!oF ziRBGv%HZRAEt*hrPI-{5{jSHvbRLV(kQ$DU!{KVohECama#npo^4#j=4BoNj;&nO4 zl#R@-%{Gh3$bU&{e1AHv18ZH=nt-Hd24)&OI{M~K)K#nx2y8OK(|nzg) zP=9|Khqt=UkGC)FT9>Ky>ifkiLMob4SAdd?m-yW@Cpxhfgp58ych13f#7EH#mI|c4 z{s3RLmGrY2zBfm>3O-|o9?ee~-=f9VNJFvAJIwm}uvtg&=8oBQCmNsFEgax`gKGnI zHCnF2l$jPSCO_9VQpvNfIM{c=CNCzm81p4bn5Ft9uX%!E)zMjEXNe~DB9PtC$OyYu zISuP912%uBY{Ib9n)ygS#Ix5#NJYiMZiV22)TB)~eb0!`miL(3nP?%w$Dg;!b6K^0 zFOiGa zmx8Fz<>68#=CvyIx&F?m`EL}rg)GNNuLRC!>Ost*a9(_%`nf}ucQb*6_SY#tGX{?w z1Vj7e4o4xZkdy{#)R#|0n^j6jf=%QsLbJJrT}@iImn{9Uw9%JK_LCg&0XBJ3$(Ui` z>46-kaBQ5@dNsCbrR+_HM7_wS{%@ld1GMkvPFxubw=|dwzoX>akNEHT@UGAICuS%3 zc+Pqg#tImlm_5yp`wO@h9#uJoaDDY^K7+!vCQ`VDC)~SW9ojcJPkJ(zs4zJ_I zCbg?hx=QmKCBa&o?b^v&+1kQA|K{_}ku#wAqEk6a$6^dTX)esA~lKGoa)bt%PpU9;f-kF9SG%lnJ}9@?^P8_Qa@ zZ5zvWExVR$S*w+B|>ph;dzmvyR;Ta(&yqwUZjkIQEksD$>SP%W?uz8e&v+=_pJ4UyCX zn+_sW<2D-vpY{&G@tr?3tLb1<-biG zm(nD$KBW&L>2ZdxSbO^omwHei6Ij2w{qsGH)G%llckcXtlNM4m zvC@09EzjfP6R5f(Hu4aFGr`?{bHPrh2k;LbRW2tU6XI67Z@J&%=j+c#ug+)*iqb7G3 zajsE^>y8qq^Du(GNl`)7nH?6tU8IT{=eNH8%3YP|Tn`)TWiKgCX5_Bvtrpz@5ix2C zOHSXZw>_SXU48E(Nx2OX$L&JKq93}=H{488%8p#e4#oC6ZM503uS-+nV#9E`YRzLB z%0_z*d6@?nHnGH#UgK)wOp0Bu_|pAUe3*qrf#B|6Hizd3QszVRZxH_=ze~-ccsMKn zlqo7Y+2b-JzSw5$Z1Tq!r6>aJ{74z8>z^s;!l|q;nQ|OoTE>8SM6d>~iPR@0t7$ID zN6Yrazcr=1~C0BrW**}5!orvKAV*JTcU0veD z(7E*W$5l}vYX8+&Ojl>Q@y&x;de|0d)dT~PAL#+N?kv>=rk`%Gifd(ZAK*?FHI8Vx zntVSx*R%C(j$?rh|Fkn;8_GC27G`^#X6~`TuBgAN`2vh0LE&sA4R~vXiYYis zBzp=0?~(EnV$J6?RNXRc3SzK}Ahr{)Wo!CVgL?QCHWWntb$-9yak+OGJ%$LL^<$9{ z0s{ZTL)rZNz@TjX?w;P2paEG>ey}>U@z%z+vaR}I@WVMl3bnEt&rfdCktEDm+D6Tu zIex1Cm&nE@o410*8cqMcy6Zdbj0VGD=Z!bB$mpzd9hp!x?^f+^0SQY^09>RU4)Jgl zxh1j>OfXu`mr3)3)j`%T7CzsbB2_8VPy}2@fO-BHs*O;r*cPIL zivJy!opm%6g*-x7`(Z*lKt;4FX|h^QPR<=bQR@U)rHY^LuMdk=OOtgH>Lw2fc|Fly zt$3~Cvq1ko2SBB|J!7KrxEyob&rzn$kzATJy#JSL0T4)jBep~DYOAZeDs+Ese=pbp zoSykQOH{yuMjh1(hb2=24%)y%Gc#;G8L9ZY!3#V=Z}rRW?UDJU&S8Gi%aO}m zjK^9Ltbn-B-h9`V4a{6$&9CEt0&Fw;{;L6A)eQFKoWhzd_+Q`~PPV+V^6DI+=Le<_ z@fwQ}xUNdVl^vEdVnf}2P~rm*jy4A-0I!{w?*Ukz0M4^O?Sj?e`xF7_Q6p@UBY@Jf zzH^O(k&NL8-}8zTAJyO5K7#XlNQZ)k?gCuiZSdg`o6|eeMjG5)dTTv1;B9B2sU23V z46$_Qz`VkCfKa88A5ik%e3Fm?bj@mU-xl zn3|u1kvh<0;hNgoLxsHqYioqb82tL#z0B15{EUo9!1uZCfZyjn(YmK=DBMVE+k7B; zzRIwzHRU$UGR|gc2PLRy4mjtC`X~tuiCAcnT^t@SHY=|hr}hGv@*OQ;z0+ruCC1Qt zT;fC+?pIaWE6Q1qXY$7`)*t4GgvaOcm>p?k@HtTog%~3ea6{?pu7p__6pkb!WB7f+ zMeqU4Py1F{C6PEfQedFowlgz=H&?z@IsvT<`vOs)l1-NgJBDCRevvchl{qgxKw*wn(ZCZ zqEI@Y&f)_ynJjbHC{Psu&lP)iX72!BB$w4a7?)r0!tVkf>E-UZezf3}%3@wYRg}c- z0N6+a>5X*NI*Pf)Se}GFs16?JDfU;#k zfQ>Xj2gFnhjN1O2`a{Q9;TuW)=EPY@OEb1|PwD1akn3q3>1A{;Sz(PiNMO|VD z+QfT?t1AgAe+rGoL$00;EIfsH>frUd`lgLdVYx10J`~j=%B}1Oq7T6JqS~#sBga($ zi}a|5AlH;;PJKz;5d{;+&No*2t?&}wTz2k`pDP7_ zKHNtcdCH?YDpv%`J)YwEA!7aU2bdCtRmN`A1y54|UsS=wux9HNkn?@{8=c~A0s!gx zGTsTPw9K!@a@z1z+CR$NTOFRn@+OdhgXX=0j0*q(wQj~q;#V1zz6FD1%n=C!e%~N) z2MyN4^%RH|&-h*y*Uwk#md3c7ymIHhyzy{Tnufn<&t!Iid3=m#M`Q!ssvZ#C&$fq9 z_x4RuMiA0|zlUwW5s)*lH5(UduwFR7ogxL4d0o00U~(w9UnhajC)U=yyB7u!yWvmB zq+${`L1jk#?1>9k?iHU+!8(jmrBFXv+I}~31HgJzY!cUkrY^)dLj+19A!wmz&vL0Wu&M0OU8o1(+veNg$m}_n}QBs<%)|=a;AG zHrWrn-v4OAf6@UW*?4;pnS}jewap7?o#YFLB2+arVBED*V|Ym!TqF+7HyOV7yxEWR zfB$i~o;o%sM_{?lOWk`7JP7^4Pe~&mgBSm>jP+{4+`tjQLf{VY_xDG(n!~<+9g(Mf zc4PQEnijY}W8OJGuL0ypSn=e-5%P;v4OLJf;`qK;#=0lpze%bQGv8|1(R!Q4TUt4vj1$)T%JI zwT%H_=s|ku2|)EzD3CyAqVv4?P5rxAPuNybJ~4pzKWuQxpz1hP&Tk-by1B6t0|GEG z@1bfAR&<-+m-(R4;!N%A;$moSPHVfo94d7;X)ZHEOg6hvW7>1J(X*4>n`_V&fc4px zuDiRN5nv^u2%#uUHwclwf>#V%%>YP@d483tK>9`d15q__B5A@p$ZPEec;iiSPHHTi z3Y|K4rr8QzQaZii$)x#wnyZz(7(<|;v%a%KK6|>_UT$w>r1TMFy-1pC6*|>{Sy==O zI!&E1X4F8@2i<>qonF6P2kIKkeS6RL04Z)qjiT@j=pVr0aUz+)VuUes`KuNy2p&{hT;tA<7Fe1wx)XHbV0G#r zE`T0)9%^Y}A-T<>qs^BM$nF9;ReJ@6Ubl!4AR?+w$v4pp)!9=FM1W~veCW_VrT)dY zY@+4lD#m2>jOX+5M8*ovQ=VeZ?9lw4fHvtC!81r5K~TN~Ec0&^ zRaZv}f}YJ$_QBHS`n^0CtCi!3vq*n<4VbKzvTB8mjg2}9p2X(V=H})ZoDT2Er4qkc zsbaQN$CJy(1RoI5ZHvmK|6~D-jcN)%u<1&sQNQvgTI=xmbUk%jjvTxA zaXE7c3W6a(G1#4SD&b{vG*kWoT=wpXlz+$w<>2GtDb$+D0$Xikz<=2ED~x%|dvJL8 zDzf7aBt_z*NEl_l1aVJXdBwI)@sj2GIt_bCS>HZ9P!64Hd*FM23eQye94cGwp42U) z{|)fTjnCG*fQ=aOD;al^COD1GtcDrTjG?`}yogNz$pq5|Fh@`=nW&#HqOe(l*yN!J z5D59`Ad*mcsXD%b>`}bdY|OyQiq^=G!*W`rfQ4)iDwE!uVYO2pMX0c$Dr+G)WVvBodX7(E<`oI2{_M-uwA1FBqqe!KIXztB!HjvR;dM{5^X|-vjQ&?AJ=)4vC1CDX zG9ZVM7}#L5ba0us&Ug+L-3_+dPW1r6jQLJ&TC&65WbPt*=jE$)VjEP7o@dF?%CEVO zbi)GK;u&0a>&g3-dMkAs*wks1KJ;LWQ~VX9A!=s=-&@0G;F4?F`Soq}$jj5(i)1(o zIdF2If7X-?b&WYis8i7bvtOS3~a~es<$s>{y8ff?;@1f(+_v3pKfO5+p!T^0%Mb0Z_7p1t!6=J%Z?&@H`JfqQ zhwz+7J&uR+(K&Mq$cX%T;2xVQ)&145TjtLD@u(D(+9z8X;12Wxrkdxm{XFI~KV;j0 zYt+F#d5X>^m(3hM&L!TTQuR3(^)4QB>_gAG$Fm+pJwPhqrF3osN%p-L>44?qsw~IK zHrS6tPB79LYfR0Szl)LSa{=+4!7nzB02|1VBnP^!`OK|3eQ|g<;&nO3(Z=I(;SQ3R ztAn}9Q+Pa1g~P*#(~eWFzFDuoY38lmE(h+rnQmF?H71gF!1YN>Tt3T8Ef z6o}nl9)vzSF)Qf%-R)rdja)w8bi8-~<&m349=l5*K>vgt*J%#qzy2mR5SC=J+8G+( z8HG>Z;Ls(}sKz%t?M9$Z|NO(%MF$K#Bez=q;*?1*Z;vAL0Qi*#CM(7s5bxN`@FUq{ z@jLIs+!{3>ON?||aY+G3c7BcY%Mhgf%ipo|LlMkQ*7VVPw9eXJ!_G>JJd3Pa$AO4^ zRKQa*nM5?=R-7c%qXZ-&rOK?Ef~g?hNb)8a7NAiLXpG6Q3fogBL?10R%JW*gXW4c< zxjGF0tOK0`&wocHx)P&4a!W!;7F_`!>h&@s?iE_y9ULMWTFU$6TJIi$1Z6*Cm3A`KE2)ZmU(TWOus z_iKZB25TY(m|4HcWd&oz48Wq@+uM_)E`OlJa#iZ z<&<#W>$rLI`P?;a^w65Io_VwCI791zmDECpCuI}P6wejIipAxS18%A1FqR<6dt%uX z<`pQR=r;yZ1dLd+BV}VK1kfiw;ifA-iqTIcc^yTOKt>X>2Um zw%?hRsFit`Lmt!Rz|kET&eVgH7320=LJ#bU6* zBzi*Njguf31{-`R_*16Bv{yJud-|i(5ou~Zs zU8MO(ITS{j3IwSTqVan~B9~at&^H4Y{nKI#0;!Jh$9Su(i*RBMNRdL?7y`Ho;E>oX zVLeL`dRYAWsvzp_q#twM3@VqodHmMUx4m_Y@C+0Fm|q(bn(*D9uIF5~q2HF`i7V#d z^Wyt#TlzZ_SH(hph!>z;*2yKBgnArKV<`eo7<(mEA7IcXJ8A1iEZ*@|nK(oT2S!Fm zxRMm-K=kQ)9oOTejn|$qiTp?n)k0bY7nu%&VKFupup=KVx+O0+ zmUfEUbS@NV1V_r%rDgNyXVNVFa!=h`%J;SoEtB@B#@7^uay*x#EMM9_!n%M{@PmfR zvd6w_h9X$#soztSY74!y_O=U>uvg3ol`#StiJEG?&d=FDIG~Wv`}S=9R1_CQ3J47u z=T9Qfy2Ef`z7H|)d9r=?nOVl*S2v&gnE`q?6huzN)zI_3GmRdArd!AV?AjY{ z2Mwx`n#cRY%EcHasSl;uhUtK|IWwPWLFe0w01fSa67ERY^w`tp%VVq{4y=2NS7A;9 zs|B9=446rJpT1dQsDI_5blws%qwEKhR339=PjIjz1F_swGoWjI+Q?pcwOPxw;CEY_4fTo3ukWR72y!L^Nh1O;|vyH zgX2Dx3iQdQsT^((N-KFNn_345yks!>9%?7}W+WDX2ZvX-_eFS?8%da}3;O)XH{w7s zw^ML&@;kz11@5UAeYIe0O^^s|ULX9C zULq1Jh6oZ5*Y`f}yVtgV0omuBS8UfcsD$SpW-5+X+m`rI-rGhZR~(%+^xbolExC#_1gc~n(|qNLKv=c^g{C*HVP*JCYJn0JL_S{8Vg z5*hVdr{_BFc?Z4razh{K<#CxqrgKvd=7I@^OD2z*}lz5KVJtyd?|)pd}IvM!>412?Nb zJ~0JocWdlu%(2SEsln0N(5Ti0q+HelL+GLl@nQe>kgXu}d4kvf@1}PWXj7bmi|M-2 z$N-KS-hEF)iPXklRSgUpt{r_E?Q`AsCW%3Z?P;|uFtnA+Y1b3zPE7an5!)#iyxxN7 znh-7<-N9|A?GFrk_=a8}Hm{0e>Tau+X)v+x3bE3GdT`(iDFi7oihp!f4=khaqx+K$ zX9s8w0-?2$D^3iy2_CP1)=xSY;`VFkv}(Pm^Wgmj&*ch3s?OfE`*(gk@!CmJKVbUj zU<=yi#!DgKiBI}iWYd^M2kI-e8^^sahA7*#?b`l$9hNt$QHV15fDl`3wmpQk|+qNo%z7uYye%K0?2yV<4p@&e~!j zC;U0SbO%ANf=H;c4$cZaP2!ls_&m7xe|3w(R{UoUKG-0wx& zw#}k2XjD{{trr*7M5URZ3%#2izvFO1H8>u({+Js}pmfW?z9I=OwnCvn#u~0FMitz4 z$`uOk)l9>2z=sFTqB#S_fB`sCB^iSvmN1o~xG;7|uDBABG!#Kxc##_Z=i9AkHD|^G zv)iyZ<{4`2w_Z2LNgdCRv~oedJ2XkAd+BykKwHUW9uv7*Z1L-jW`Ts_n7=}kJ+Yr% z_d8s6+vNi2h=0wFj;#=s8;W=0oT*9E6i~=01iAS0Dr(PN;^z>l zkRSMb$W%mdvL=t!N$UUe_*Ghx)<`y+$ngx{%#b(?3gBltUY<+erD#SCsuI9AO@s1; zT%P|av4b+CS7-CIjYCZ=3aetPzx=JtXPIn1UBJMm;aKrbaJE=mUBhKTXDH!M!zB8h z&+PjcI-PHhh|O=C(Lw1*#bLr>(&|@W&vyrN2Jx+#R_RQQNerjaU8xowXFryezUuDF z_82*rq|mo6y4=UrqW3ja8{f@vHb#FuoCcn33$F7`ex_j{uSajANF^3g&pTMjhz5&z4hH0lg6YZt9({)^*R8HbVM z=f0;9AJR$zI@{+r5l2fM^0p2xUeg6b4#P~d3=W8p2#-JZIY>WvPaVK8vAvjVJq_>GaYC^545q+SvMp~Ok5F=c4Q zG|;3v&nP@T4adu~$YjeE?vaB~%oRCO>dlixLGJuToZGGI#w2&85oGP2$ho$gW}bD; z(6s)}r$}ru074X0GtroLZ9W4C6VQY6ROfHTjbTQK78D8;Vz5pI74j`|GPom`AHdZN zPEeO~J18ptHNufHPm4Mu?0<0B3i-5AF#|Tu&mYG$l^wfSCyh#_iNV+Q_Aib$MwUX& zA8ojE4~JOvn(PhzXg`j3jrFrZ`{TU(ag)p_lUm`}WsU+d1*K zkB42<#N152tjXeUPD6%dF&G}@Y@2t?uHoLyWUabB9Jf3#`M;C#E zJija0EqH{O&l3^rlSwj(|cQ#;^n|C1}uqUhE4dn$aoi0gNmoRHO( zx9C!KqsWbZws>{}o_6W7>NL8Y~)**gKtSFsV zsd&3uCxN(pD)yv0Ll4q?7ZZFBGL&zi#;Rq<9UU>>&zvDU4EvpktFWW`LP$Zp`~IvO zBFU6;{T`7gNAYI{D)nfE@Eh#6T2lgBB5$x0k{IpSeP$jJRJ3pqaXwQ#6LByqE>lUd znruWU7OWo>Ok_^v9d6#wmapu}$9w4e(c3I86Prq<#DSW05Y))UC!VCo{zT-H->`R5 zdA8ikTwRK#=NOoQrY2bum*b(bcz*D`NZbG1%He?u=E?L%5?B=&&~+bV3Ie|&>RY@Y zCwsl{U$!O9jgYs{wqO*#e{e%jbCRGCAo+S0yajhP5l0$o_6df_Qs|pQsQ+`5wy>_9 z$t|%du`R2PYm+S($JZ!rxvz(O81b}06f_$xAjjmLC6Don3Zj^-(ETj?2F0@j3>r(9 z@FcphzJ~=}4mvD@ftKGy_{R5-@Hl(s!M<_@_VCz=gTgUh`06CE3zLJ++c;`=p!1>! z)BucFh%)I4gvr#osgZ_u_-vOs?Lb56S&@`p*n@O}uSA0X$vMQ;XL3jArz}e9q^VFQ z5%y4(Vb5b}UZ_fo30J;9_bc(5CXz|>laVM>bQ4^M@}$~g6}EjUm> ztjloY?yJ|#8R##`HzuG?i(i~XU43wgKhiScDAg^BnJbi5AESNI*d5o4{?5i$PjS6% zlJl5Mr6@$15sr}+J8D`mV>4fs3nZiP${%{|CJ`7tkhEtVJ*CTf+8o~~A0%TZ5>Fl%BlGxMM%b0NmHeky) zHil0XNeP=^u?=K(W5nrks9q*`E~8*-I^)HdD%2~pU_K>!nPI~^dW(05L$G&1OTA%% zk8jLY$p*lPYX*=u%yWm`yy7{w-YKYdu)!S|0-Y{+SXx9Dw4oh%j!Xc0{r3w)N; z4!uF@ie`>!j7{tl7@k@8{!5U9zm-)dt0Vn{4v95$!e&9Ouy}`b;Q@|Qn19gk1`dhNumww+9Fq_|&+;}ZUo|#ZM zUvulVbZ0CS;VSHK$~`anBQG*^U-SE8tOuzdRHRRWJ_S*6LtKBNrOhiEJ%bl{MtxL-B93^%aH+allp*fkHoDPq57!MtZFKWXY05Weey6+Dn+W& zp!cFOh@z5`C>CJ_gLSj=;AjWMk`ky z50Fvvd2@h)J@5E;6k8iJ)zh2`Qvt)jK)q1^Z&GE7WQboTk=1@2h^VMm<=H4( zVn>LOBI(2ibAkpm8N9HO()$ooWWkt!kqw?ah@Ig%N`}Zt8l@G4^TPs9KAn1O%Aa)M{3qKO)nYe6#%q#`rA8a39z<`HXA+FK!Z_$c7 zFHogrDu69g6t5k^MncGArew*RJ=qtu0w3E3);VE?S<`yS@M_TFWfA#FqqgqZCF_W*H2($5C2iuFCRd5?K%>GL)^;B zRbU!ghp?E2%gs>~JQQ{6ug^r=AnbJ7tPp-{kilkLG}IzS!5L!1^msTJN7|B;`b{nF zG&WhL@ENpzRj9kkG4)CkV0GNU1@3Zmv%NAeBfgJ1Oyq)sG@m%K{qgP4w~|l-Pl-cJ zDDH;D9SHG>?H6&YQ^BI=!# zz|j7B4D*tWt{2fem9dwtBpoav|YXe7NAU&NMhXk4%TqsJdU(4k(xh=n84`7Gj+#h8~SJC%qb5XLsf!A=8Y9$DL zdEDhcuNh$aok(YgMBt3UcmImD;)X-xOtB8`+C5*RUIX@|cbhzc$b^SIa;BVur7gqu7oKun~qXrzhlRzQo~|mU@*X*c-I*RSU^2;=|W3CKdilcVN_U`B7Kjc z7p=VE%b~rL$!<|9(ax!zJj+of7*`(oh z7ez^jk*$Ngi;V~hd}GhA5%^bY;_1HA5>H6_)nG_XmGn^IpBr9889Yl-bXLE)vU+-Amk=is3?KtffY2*)m5Te>Yaj+qVG;6 zVm8u)3hO&2%#i;T0LwFYCLw^mqMLiO)iS~VqH`+D$e=lXtCMKn&VS zI!|;`bMhMU$aq8N8TD4mZK>oyO&T_}lU)fN6xJN9>a2Sj8L>PUVf_K$w)p1#TGlVC zSQS%|RWu%{CLcNcoxCMZqnHKCWZw9uiP?7T}^$JL;a+jkZo+$ZS-ncsb2)V7$+4fzj%VxZF4x&Hp*D|N3i~%kVr_+LI%044J z=^c?&a`PDSa>?88@0j;p_NU2R=W`q`3}i3TvqqQCLf(R<;A-`)$v|#SBlU8%^zB2B z=gKdH;=#U^O)1kL9#OBHCZ>&=kcCG+27?8)KKNXvm%PXViLSkYc^rp@Uv}A}SvU2` z&?Jj?%{v$`VTK~H&_|#Kc?13236pv`B~pUQ%swN*E&lSTT+wnVmsV@gPSN6VH6{K; zN{!jSQvoK_1Z}6j`*ZbHHBrQJr<1J+c4!dh68& z*_P|mPxxN{5=oKABiJl@q5O`wv3R~DK3?CfJtrF$Sc5lX`CW7*W#k{kkB_Y5c!V-( zBx{vA)P(T(G_lK0vCd7Ga@g0KkxK2(mL8kIm?SkKMWBh);p!;!axubsxWP0{0DckL zTS2XC@8{-G{Th*zlZ&S1nfl04?K+8AVtY*PuE=`tIpbKVhoY?bLaW(5ruCM5;>!!$ zm3G~B*Eu<-==1JE>7AYuWs`r8ch8DjG68I#do}BQez_#T5f3qs^BvMzuKkFn(`t}u zI9?|Ib#t8ZspDyuAE&42#K(52UOR;I(0inzoY{RF%57DTo+W+dv!Cgym z%BH7>tMAw8>?1e6?B!p>vwdkoGs$0Oy-tVv$I^FPZg@JZ7d8pmq7oyatw}|{cKSN5 z!(QCA-c46(yLptYJYB5M$d^{FHrb>*d;ZhB1I`~F=lcZH)i&Du%fC|SE-7~ipSvJE zeLC-Dad%KDmtl7&J~cb-BpL<7O8d8ZJmvSNnJkTG#`9XKQKxo3aGowZ6&kLz)EJzT zEVNstcyB{kQRP#5Q$ZpSBsvyKN&b~dTX~s|KW~L&w)to6F`pC*`%@*Z5wevCQ>~JL z12{h7ae1n!RcKS{2SY#4iN_{)Jl~0f**7sL@RMr_rKEry_EUaHoFh2Y1z5}(URlm< zFMLDUl(8xd+G5|?Ea{}xDyEG+oOa@4+?I1+XjAA9gYHo;hR+(WHH?0jX{q8~d)6}~ z{zRv}_ljm2T77zOY4tOD8`JFcH?a*8;a2;6D3Lgj)oNMeqJ`N}HGv|2WrEC2HD7Fc z9@?uReB4gqSA*7exWft8?jU-K4$0l+-vnA*HrE%~ToxtJLySfft|k%PY?7?yDy7_; z;NiC()QlcZGX2v+_(V@y|F!!%9d$7LS{aoqYYzI@BG(&2L5?BE`_W>jM?(8GTz=>v zd|c6w+&G)!Cwmn+lG`Grd4qr-PB%Tz`Gm%xAb>!?N4MnGhX_&N=XKJ!_EQ|9JNQ1A zRQ$32_M1ErTHcuOi1-`igQj6Hb~AdnB!TYD{UE)7xk>Y6f1uLbJuwyS<+Pi{+k%ii zk-l5mXW&t-jBusQ?>?tHU1&#m$3muXrcbG&kAD|~I$K{L;r|#4r;dTI7z5w-Kw zw#NHzBW{Jz`$E!M31#)BK$E3V^65UFS9|%P%wHGYK>6|2P}um+vf7m zJWqA4oYtw%KxmOKi%Xcm5uSh9ag5DTztQ&M;;b=~XpeTGW;e_w8z6WzvyRr}SAG4; zY;n!A(zO~ABR`dqzqFn5VW0g?zhAZWx;0#QJY?TrX|8Smg%S*F*}xV6uaC{A({fEn z1b;AleM%HhtIgiruKRhR&Zg8BlR-t}ZeialLmqG2}cs$+`UsU<^VF2WiQtmBm#)h}w%TpuF>e@=} z&Ul>g7T&!0!q9i}$SdzV)8-0I7V|XBZ>Q>I-(YrzB{kK51bX4~U4(i!T9@bz__&@d zeG0zV`NQnb<~z8O!t7F^pB0}@|JhZdqy=;T$OaOTAZpCp3~R~=qrpKeH@7!F&jB8N zwo|9uUm?W0!ok+@yiX|w`c(Lzlg+-7Xkp(@iu8GRFk3v9AwJ$#%M5*4G-?7Zxb57h zL>*PPD7Z&yRW#neYv>Bblp;0PRA%R-gpv|zLa`yBdBp0Ud#o8L&M06X@+yqi(^m+C=E1vjmb6UsJho^ z*B4lt;7}hDhXjB4J5H#?pxvS8zuOwXh{R|*AsT>_oMFK`@G+7`>)Lih^=@nUc7`q8 z7O8jFbIf+uxmv&_gHUO zdSoF9Z2soAQa`fB&BboLN~hgmchk5jB#b}$oJ1-v$8~trgghiRZJ<^LfXDr`kokI^ zlKI!9!G>DW@s@N~YjRxLRVE-w&%Tor6mcLm;sLPDXT2VYI}k?V;o$W?&QXOjz{Lu3 zGXMB~KB3V%`t$mL%TudFN03; zVbsG;yO<6Z4!RzK29}m6mjLI*k}s2Ador|dPyI>eXvSype}rM_N>>1{l5Cg4ko2kJ zP7DD*COzVc<0i{}PRn=s7vj3_XV=rX{aW*p2{}txKk*Ji?t+<82EArxx!GSzqLBwp z*6Ynwh-=<2ZONKtS8q-Rq8yg9Ya}!cn4)H)$>k(>$=DoV(n`2joML~M5b$?>q!wA? zdv2oWE>@~2vw}#GDnTKy>nhjySx~0iAvocc?`DV1PGxKSGCtuG^qfV>Uz= zeK4EAKBr4Lllhg9)jAbsH`D7TjM98ay@L4Jvm3&4sk!#*cve@f@Ov_6SCYiZ0wp)*B8y^SIJSIKU7rawgDNVw&` zdm$drh!3aG6guhMIIBG*!Q&1ynE$a`Zcz=nASOWQW?l2$byT!f`4q3<|6 zdZC6=J~v(w@Mw(e^s#*zR4Ve$I?O8n@!)gPajf>DO+YA}VMK=cnh8^qRsp2{HclPS za~2l&Gr&tZ%&qHruhcW!b&GR4T9PApSbZ%=y+*J9JCmR+igla$NxUB_8OSy8$>ChvWD)RucsKtL<$k z1NUu5$?1hXnQlX}dn-ZsJzH4r?G#jQ_eE_A=D>2{@}4j zKey}^3kSw=p|@SB;e;dYYyU>*)%EE)T=s~vIX7?;<5EHpQ%bV^C|F)@Ds+5qR+x*$ z5la+W>Ij6-^bd#kKALxh!W=oK=+U5twv|WV^4uJhDpiU5#JlJtLmuS{qAi0 zaJA}G4B?v;wK?pFv&S;?xpF4B9_X+yH$pFdp#G$TM8uD;Y)?B){@|EkRrY=`=nn|v zE>FABxP?-87uDT3TAZaZtjQZXv+MEmN1CjIcTT(0U&)j-ne`n73Ykp{l$tM|laYsm zclD7f@s{WFf(b}B<_ld8_qn%$wjkv#)Z|^|&GDb>FG;l0^)O8DP*X{q1}bz9Vpgl2 zQe<3*)f5mKY+h7(DMm8;yd*QZ!B^Uz@#DrKl-N{HzCX~j_ecAr_Ir(6OtDBpt=#_T zDVa`3LwCRFT${_3WH3rBq%9`{yMMnQq9M)%W_yIPPp1c?1wt8Tx&t3(X`_VleU16Z z-^#vkt%5|T#iy30UwMp<4u-5$Lhb7m<*wDBw3y7B`TtI zL!$-1@ez~XUh{hnH;4WZuo4E#VilYc*xGEY)YGER+Ew!nX`#j}Qdw-=^>8=KsR3qO z2Aidq8Zv1(IY}H00lVB}Z9N5uc?MQUud>R_hmo9k(0HVC3t5^%ymIVAjCMLXl|j58 z=e)$EznU{5`c@WE>-tlztXMd(!WCnj?d2|(0f5wFt8+5@+D&#~ULsOEb4*XTsnmZ) zl|60%VL7Jt0exS^Mh`Ng`H|0&oEq!a;OqhR) ziEA-OW83&wxHe`Qar*bWUl;lBC#{0G+T4-6osp;fxY~n(?Cq7Igg?PUsn!9bi#l#? zX!HsK^!5HdtMN}2xqVLy=7R<U^3#F!S zuIEhG>^g=HvVG&9?OHcZ&>K6R+@0HR`1WTTD_dVT>qedBg?lz14O!H!^+h4K|O`ib%v!%(c%IZ{ap~^JU8O)St{%oG~V-ei1 zdTIG$d?tRLoxdR~t)0bIbCsQWk=WhMp1Ygouz64=IQ}CYff;>@A|EBaY>vn2E|G1u zh^a3z_x!Y}aTMaf&9#di+F@HDjKFEg0f<6FTM)(%pHCzvZ8DIqAWvPQcR^{gMR$e0 zSuwg#x8X?a6Uy3>Bn&}dt8$;V@6~J6{e^(XG3c?yDdp;TQ8ogDrWl@7ag{cs{;|JU zvBtLNxSsdM(&!>7$ve*d7v2>~p~i6oHYdED)^JZ;@)7xtN|@45YW0KbXG^CX;eJ!^lb zF?9diz$@fvliLhyLnG0hdsk*6G*bzCJLn_)oviO0RK?9$o|_CAzg_&57>IPl&)1hD zv*j08o&LpT2ANq$KWu6WJYs-KclmoL9;ce?9Rxw z>~J~^+GM7)sN*FWZ=qKF*e7T0ap?I;DLOclUYnh3yZv-ImSVV%Shx~Sv5|Y0X@#21 zJ3QpNB1W~Tz+ZE|U(y)Em75Ys?Js_{UTqOI8@;2EtF!S8eU`b`y4)T{PpbOMLAJ7g zTcYgz%lqy(lkY}rTp*i?=_5@Pd6@^)vL39HW_echDA>9 zPXNJ|IH%_&`@t>x=O0kF4pQ1MZVE?h-Hd!(sP}5rQUxP~-y0UxnRe}WVgLqQX&Mmb z9x|Txm5Xck=`o>9qgE`bLo$IvvQ(}7ku(IEm!b+j6Db!AUM|KzpZ2Cbc5`y2_8nYK z@TdG#Exq?rVrTsT2JW*pXXc5*HBr69ileZfT8R*OufD-Uv&g}q&W8vw&4ajiv{)}a zr%dDbcpgfpQ6=!5)pGMxGR?%~R;fg(q^2X@cNPqO53AY|M*)zpFqp6}TWKhrqY~&K z)2q5`L?wc*BB| z8~|{6u{mrr{6gXga++zBe?%n*f-gOo`LB5aXkt+;-jo0~_&q7|d|d3_ z-(d{!B4w5ot{rOy% z=b4%3cQ3!+{r%kcUx%`&R|t4huoX+(f%$bx`~SF*BWF{7>o9#d=#3Mo+9{4XMlhc* zXW|)1f0>dE*mc5$`i2jyO%cZ85d5D1v4Q+A7oIorHK@JpZrp(C(Aw6A<<8`HT`HRO zn52l$d#R-u7`tX&9avWDGrbC8_-i0uWIpmdsH@xc zusDGd)B@CXpw&;@03%{PLFWSHvjoD{HQN?_czE*W-#7Iwc8RzKlhEYa_4jmkQ<1~k zL9|BKp?lQqNZkFX4xvEFt@jjAsk|s(i244dCZvkBf?lU#rej;azv!dwcV%(iP%w@@dEY?)+uUz#$qnzv3qlaSh1htSdaSf%rY&6fdO+LQrzk2B02+0(Pa~X?jKS2(AS2 z1-B23v4_#LOD7lVIC@_g3G8~rBYHGJK!;HW+Rs)?sMW4iFY*W6=+dSWYEH%}J9q0+ z`~bHZhetgE3prKkJtcI+sdkfQE3cpJAu+c)k_psc!i}sD%X#kLCU`XpVCbp9KwzO5 zJr25kvLilG#$#6##!Q`qi~pynBCcR28EsAj$a7YG21~5xc6QpxNwk_rHDm`2d; zzwPlT{u0sc@;p$4l^rXr7hl9>LQw1RV#kp)K2Pw$1P*e&bzr9OIP*~+z35za#4s6q zSE1_1@hn3kjzdE0)k))cT>E%zT^N|?i_Lwi!f!$2(@Xf`sf(F}#-DfOtcL=FIw zCS%2w&yX{OMykOwDkbZDpoF?a)OrJNxr81Gf{Nw=T&Z&+`Sm5m&^An`7Ffm&&Vosh z5RfTCp0Pw5H8u$i@9y8n5J$_Ci#Z7praT|$6}V;uEeC|Z?#JR_W{b$==0OA6 zj{&NWkHL=qJFw^7EJPJm(`*8O4cRFC?s+B*xVq#0hB$D^Q^VC&MSD)4f%6;B(}!$I z^J_&pB2JVpHu;E|ftvRkuqvP@9N{-|et{YSEMeL=uSYdJ#N|(L>Ar>sw(EHgmf!YF zdF%XaBeH7n{ceTD962`cM3z{vZQ#7Q_RFuzDx4iyW2>7=76)Kz)B-uq6L44P1X2rf z2KEFY4%c{{PH=L6wkg&Ygd{7qk8X~fUh)9uptb$UiOoXB&My~%(rkHr>_shF0lkBv z;lJ>#)|WR0v6d$+{1#owA~ckfLV$zo1zAw62f9hWXmF5^;2)M!pf^i}YSF|~N&vaA8uq91ea$@^6||HY9t5jH$!s?kyn#1EjP8_? z@0%HaTeS5&e3H#~eAITwSH`h@q$Jt1N)FH*d)p`ZScE+8Lq(`k?x_r&kV?CdmCw+m z*kIQE;305)zq!aeEGB%H@yPCF2!N%$%j?M+08#UoeVW1iAJa!2k<6<38rwbN@j7eR zC}gZQ*^y0D?gDoH2061^X*cI?XJ9SCBsPQDCYU7BW_`t&N< zhKTfgR{Qi$I0-ML>$hE>c!&de+wO;VK!UM6g6s=Z#Kwk%(^U0|(iQ0XYrC%mmuzoc znH^O1haSU1+$CcpS^;$OfF*MoSeD62l?tq*ee#7s{ng10lN(XG%vM+aOgd6 zwTpq9ltA2lTFo&XrvH z-LpM9e5d1vxAdnc{O>?tlino(gRvlSG(|R1l1TU7c3wr6+S3#)?(#!Q@xz{CaEH+Ciznp;sc`CPc+?&f)mhoFas#Qvy z%QX?(G_P@g+c$b0%_DC85vyLw8W3%HSX4FZ7|6s1Dm)ztL2T-+yN_B)13T8YzE?mC z5q>t)_)3K;Nki$L*!xD~?ABWzPEOPf>Dilzu4YfsIgTM$=m8VRM~QO9uIb~}`wAmb zH>FsK%K_x8lN>YR4$Y-GIf1gNX!c~0?C^8?Bii&KFS(q~lUu>2Y;A-|yJphR=ZZo> z22w3jrO1;rs93_JzrI?`Q6h&cL*;flOrTPV3!xE-;31c#coQ@*hwdNkR4f295IOQc9|;=`CXfR?){`>m-RH$vb{Qx zJ~s3b`_aeNeAQ-1dgEVG*e(?kJ>X|M-kI~8#Y1G-{Q48>6 zfEjj%2I8|K`t?u*r3!}BRdzB#;XfT%2OdW&56iH{Cu%*8f@K(iG3H4wM!dV$k)f&T z`%Rwgn}5*;7*}l^08uSzidjV*mbt=L7l&HW>^u@_AX}6idT6Yx!x-TxbFnRT7M#iD ziqn*>qntsp_v)daa@Xymz?0PGJm*;0KM~(*s5i>>gkQtm?Q1}W%xj3kr)R(W+X%9@w%Sey8c`g?R|{7p+QWN@UOLmg+ahn zq->=C^-=8=VaWtX@F&bmYXO)IrhFon%YqEvA=;k*q0jXvaB?io`_s0F)+MEv+ zPY`V#Z%L+B6zq>e93R?%Ao7#m$l2}CW29eEW-5Id=J9OEH(({fC*c#NSxG5tzoZdu z?s2|P@nLbJvbKF4B6jad>!r!AH=;rIvVE7ng06L-%J0n%Qr6uyWBK)`Zk5KDmt#if zz(bzCPSrz~@CZ^^K?o0kSw9zy??-*TQ zuxoB!kPp}JQA(p}uS44wKw?wX;RqrC4X=&%efrpOLfBjMXWolj7?VQpsou3gySu9s z>Gkr5&qO*L!2en_6Ogy10+5}*^Xx-=F{=8A=5__2^hV9O3jJmmF77MO+1Si! Date: Sun, 2 Jun 2024 16:22:50 +0530 Subject: [PATCH 15/18] Update transformers.md Added image location --- contrib/machine-learning/transformers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/machine-learning/transformers.md b/contrib/machine-learning/transformers.md index 0799b7f..0dcf697 100644 --- a/contrib/machine-learning/transformers.md +++ b/contrib/machine-learning/transformers.md @@ -7,7 +7,7 @@ Transformers are a revolutionary approach to natural language processing (NLP). ## Model Architecture

- Model Architecture
+ Model Architecture
Source: Attention Is All You Need - Figure 1

From c6c8ffe1f694f2795c468afac8655059146a2d1c Mon Sep 17 00:00:00 2001 From: Ankit Mahato Date: Thu, 6 Jun 2024 06:01:22 +0530 Subject: [PATCH 16/18] Update transformers.md --- contrib/machine-learning/transformers.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/contrib/machine-learning/transformers.md b/contrib/machine-learning/transformers.md index 0dcf697..7d4fa38 100644 --- a/contrib/machine-learning/transformers.md +++ b/contrib/machine-learning/transformers.md @@ -6,10 +6,12 @@ mechanism. Before transformers, predecessors of attention mechanism were added t Transformers are a revolutionary approach to natural language processing (NLP). Unlike older models, they excel at understanding long-range connections between words. This "attention" mechanism lets them grasp the context of a sentence, making them powerful for tasks like machine translation, text summarization, and question answering. Introduced in 2017, transformers are now the backbone of many large language models, including tools you might use every day. Their ability to handle complex relationships in language is fueling advancements in AI across various fields. ## Model Architecture -

- Model Architecture
-Source: Attention Is All You Need - Figure 1 -

+ +![Model Architecture](assets/transformer-architecture.png) + + +Source: [Attention Is All You Need](https://arxiv.org/pdf/1706.03762) + ### Encoder The encoder is composed of a stack of identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, positionwise fully connected feed-forward network. Each encoder consists of two major components: a self-attention mechanism and a feed-forward neural network. The self-attention mechanism accepts input encodings from the previous encoder and weights their relevance to each other to generate output encodings. The feed-forward neural network further processes each output encoding individually. These output encodings are then passed to the next encoder as its input, as well as to the decoders. From 3d52c978c52da5c0120cb73b4be8ee77d9810613 Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Thu, 6 Jun 2024 13:56:27 +0530 Subject: [PATCH 17/18] Update transformers.md Replaced html with latex equation outside the blockquotes --- contrib/machine-learning/transformers.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/contrib/machine-learning/transformers.md b/contrib/machine-learning/transformers.md index 7d4fa38..95ccada 100644 --- a/contrib/machine-learning/transformers.md +++ b/contrib/machine-learning/transformers.md @@ -9,7 +9,6 @@ Transformers are a revolutionary approach to natural language processing (NLP). ![Model Architecture](assets/transformer-architecture.png) - Source: [Attention Is All You Need](https://arxiv.org/pdf/1706.03762) @@ -21,12 +20,12 @@ The decoder is also composed of a stack of identical layers. In addition to the ### Attention #### Scaled Dot-Product Attention -The input consists of queries and keys of dimension dk, and values of dimension dv. We compute the dot products of the query with all keys, divide each by √dk, and apply a softmax function to obtain the weights on the values. +The input consists of queries and keys of dimension $d_k$ , and values of dimension $d_v$. We compute the dot products of the query with all keys, divide each by $\sqrt d_k$ , and apply a softmax function to obtain the weights on the values. > Attention(Q, K, V) = softmax(QKT / √dk) * V #### Multi-Head Attention -Instead of performing a single attention function with dmodel-dimensional keys, values and queries, it is beneficial to linearly project the queries, keys and values h times with different, learned linear projections to dk, dk and dv dimensions, respectively. +Instead of performing a single attention function with $d_{model}$-dimensional keys, values and queries, it is beneficial to linearly project the queries, keys and values h times with different, learned linear projections to $d_k$ , $d_k$ and $d_v$ dimensions, respectively. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this. @@ -41,7 +40,7 @@ where the projections are parameter matrices. #### Masked Attention It may be necessary to cut out attention links between some word-pairs. For example, the decoder for token position -𝑡 should not have access to token position 𝑡+1. +$t$ should not have access to token position $t+1$. > MaskedAttention(Q, K, V) = softmax(M + (QKT / √dk)) * V From b230028124bcc03f56ed8f87ebeb437a38bce21c Mon Sep 17 00:00:00 2001 From: SAM <60264918+SAM-DEV007@users.noreply.github.com> Date: Thu, 6 Jun 2024 14:17:40 +0530 Subject: [PATCH 18/18] Update transformers.md Mathematical equations are converted to LaTex mathematical equations. --- contrib/machine-learning/transformers.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/contrib/machine-learning/transformers.md b/contrib/machine-learning/transformers.md index 95ccada..5a27688 100644 --- a/contrib/machine-learning/transformers.md +++ b/contrib/machine-learning/transformers.md @@ -20,9 +20,9 @@ The decoder is also composed of a stack of identical layers. In addition to the ### Attention #### Scaled Dot-Product Attention -The input consists of queries and keys of dimension $d_k$ , and values of dimension $d_v$. We compute the dot products of the query with all keys, divide each by $\sqrt d_k$ , and apply a softmax function to obtain the weights on the values. +The input consists of queries and keys of dimension $d_k$ , and values of dimension $d_v$. We compute the dot products of the query with all keys, divide each by $\sqrt {d_k}$ , and apply a softmax function to obtain the weights on the values. -> Attention(Q, K, V) = softmax(QKT / √dk) * V +$$Attention(Q, K, V) = softmax(\dfrac{QK^T}{\sqrt{d_k}}) \times V$$ #### Multi-Head Attention Instead of performing a single attention function with $d_{model}$-dimensional keys, values and queries, it is beneficial to linearly project the queries, keys and values h times with different, learned linear projections to $d_k$ , $d_k$ and $d_v$ dimensions, respectively. @@ -30,11 +30,11 @@ Instead of performing a single attention function with $d_{model}$-dimensional k Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this. -> MultiHead(Q, K, V) = Concat(head1, ..., headh) * WO +$$MultiHead(Q, K, V) = Concat(head_1, _{...}, head_h) \times W^O$$ where, -> headi = Attention(Q * WiQ, K * WiK, V * WiV) +$$head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)$$ where the projections are parameter matrices. @@ -42,20 +42,22 @@ where the projections are parameter matrices. It may be necessary to cut out attention links between some word-pairs. For example, the decoder for token position $t$ should not have access to token position $t+1$. -> MaskedAttention(Q, K, V) = softmax(M + (QKT / √dk)) * V +$$MaskedAttention(Q, K, V) = softmax(M + \dfrac{QK^T}{\sqrt{d_k}}) \times V$$ ### Feed-Forward Network Each of the layers in the encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between. -> FFN(x) = (max(0, (x * W1) + b1) * W2) + b2 + +$$FFN(x) = max(0, xW_1 + b_1)W_2 + b_2$$ ### Positional Encoding A positional encoding is a fixed-size vector representation that encapsulates the relative positions of tokens within a target sequence: it provides the transformer model with information about where the words are in the input sequence. The sine and cosine functions of different frequencies: -> PE(pos,2i) = sin(pos/100002i/dmodel) -> PE(pos,2i+1) = cos(pos/100002i/dmodel) +$$PE(pos,2i) = \sin({\dfrac{pos}{10000^{\dfrac{2i}{d_{model}}}}})$$ + +$$PE(pos,2i) = \cos({\dfrac{pos}{10000^{\dfrac{2i}{d_{model}}}}})$$ ## Implementation ### Theory